Interactions between vehicle and teleoperations system

ABSTRACT

A method for autonomously operating a driverless vehicle along a path between a first geographic location and a destination may include receiving communication signals from the driverless vehicle. The communication signals may include sensor data from the driverless vehicle and data indicating occurrence of an event associated with the path. The communication signals may also include data indicating that a confidence level associated with the path is less than a threshold confidence level due to the event. The method may also include determining, via a teleoperations system, a level of guidance to provide the driverless vehicle based on data associated with the communication signals, and transmitting teleoperations signals to the driverless vehicle. The teleoperations signals may include guidance to operate the driverless vehicle according to the determined level of guidance, so that a vehicle controller maneuvers the driverless vehicle to avoid, travel around, or pass through the event.

This Application is a continuation of, and claims priority to, U.S.patent application Ser. No. 16/824,583, filed on Mar. 30, 2020, nowknown as U.S. Pat. No. 11,307,576, issued on Apr. 19, 2022, which is acontinuation of U.S. patent application Ser. No. 15/644,310, filed onJul. 7, 2017, now known as U.S. Pat. No. 10,606,256, issued on Mar. 31,2020, which is incorporated herein by reference.

BACKGROUND

Vehicles may be used to transport people between different places.Normal driving procedures may include maneuvering the vehicle within theconfines of a lane, maneuvering around turns in the road, and safelypassing through intersections, as well as complying with traffic laws.However, during transit on a road along a route between two places, avehicle may encounter an event that interrupts normal drivingprocedures, such as events that are either unpredictable in nature, posesafety concerns, or require responses to spontaneous visual cues ordirection, such as hand signals provided by a police officer or aconstruction worker directing traffic. In some instances, due to thenature of the events and the potential for adverse impact on traveltime, avoiding such events may be desirable.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentify the figure in which the reference number first appears. Thesame reference numbers in different figures indicate similar oridentical items.

FIG. 1 is a schematic diagram of an example environment through which anexample vehicle travels along a road of a road network.

FIG. 2 is a block diagram of including example vehicle systemsarchitecture and teleoperations system.

FIG. 3 is a block diagram of an example teleoperations systemarchitecture.

FIG. 4 is a schematic perspective view of an example teleoperationssystem interface.

FIG. 5A is an example user interface (UI) to facilitate interactionbetween a teleoperator and an example vehicle in a first example eventscenario in which an example static object is in the road.

FIG. 5B is an example UI to facilitate interaction between ateleoperator and the example vehicle in the event scenario shown in FIG.5A during example interaction between the teleoperator and the vehicle.

FIG. 6A is an example UI to facilitate interaction between ateleoperator and an example vehicle in a second example event scenarioin which an example dynamic object is in the road.

FIG. 6B is an example UI to facilitate interaction between ateleoperator and the example vehicle in the event scenario shown in FIG.6A during example interaction between the teleoperator and the vehicle.

FIG. 7A is an example UI to facilitate interaction between ateleoperator and an example vehicle in a third example event scenario inwhich the vehicle has encountered an example construction zone thatincludes both example static and dynamic objects in the road.

FIG. 7B is an example UI to facilitate interaction between ateleoperator and the example vehicle in the event scenario shown in FIG.7A during example interaction between the teleoperator and the vehicle.

FIG. 8A is an example UI to facilitate interaction between ateleoperator and an example vehicle in a fourth example event scenarioin which an example static object is in the road.

FIG. 8B is an example UI to facilitate interaction between ateleoperator and the example vehicle in the event scenario shown in FIG.8A during example interaction between the teleoperator and the vehicle.

FIG. 9A is an example UI to facilitate interaction between ateleoperator and an example vehicle when the example vehicle encountersanother vehicle parked partially in the road with a potentially dynamicobject also in the road.

FIG. 9B is an example UI to facilitate interaction between ateleoperator and the example vehicle in the event scenario shown in FIG.9A during example interaction between the teleoperator and the vehicle.

FIG. 9C is an example UI to facilitate interaction between ateleoperator and the example vehicle in the event scenario shown inFIGS. 9A and 9B during example interaction between the teleoperator andthe vehicle.

FIG. 10 is a schematic overhead view of an example road networkincluding several example vehicles en route between respective firstgeographic areas and respective destinations at second geographic areas.

FIG. 11 is a flow diagram of an example process for operating adriverless vehicle in an example driving corridor.

FIG. 12 is a flow diagram of an example process for operating adriverless vehicle according to example first and second operating modesin respective first and second geographic areas.

FIG. 13 is a flow diagram of an example process for operating at least asubset of driverless vehicles of a fleet of driverless vehiclesaccording to a second operating mode.

FIG. 14 is a flow diagram of an example process for operating adriverless vehicle according to a determined level of guidance.

FIG. 15 is a flow diagram of an example process for operating adriverless vehicle according to changing levels of guidance provided bya teleoperator.

FIG. 16 is a flow diagram of an example process for operating aplurality of driverless vehicle according to changing levels of guidanceprovided by a teleoperator.

DETAILED DESCRIPTION

A vehicle traveling on a road of a road network according to a routefrom first location to a destination at a second location may encounterevents along the route that are unpredictable in nature, pose safetyconcerns, or require responses to spontaneous visual cues or directionfrom, for example, police officers or construction workers. In suchcircumstances, a driverless vehicle autonomously traveling along theroute and encountering such events may reduce its travel speed or cometo a stop due to, for example, potential safety concerns related to theevent or a lack of sufficient information to continue traveling alongthe route.

This disclosure is generally directed to facilitating interactionbetween a vehicle, such as a driverless vehicle, and a remotely locatedteleoperations system. In some examples, the vehicle may be abi-directional vehicle and may operate generally with equal performancecharacteristics in all directions, for example, such that a first end ofthe vehicle is a front end of the vehicle when travelling in a firstdirection, and such that the first end becomes a rear end of the vehiclewhen traveling in the opposite, second direction. In some examples, theteleoperations system may provide guidance and information to adriverless vehicle when the driverless vehicle encounters an event, sothat the driverless vehicle will be able to avoid, maneuver around,and/or pass through the area associated with the event. The driverlessvehicle may be configured to send communication signals to the remotelylocated teleoperations system, and based at least in part on thecommunication signals, the teleoperations system may provide thedriverless vehicle with guidance, including instructions, proposedactions or maneuvers for the evaluation and/or execution by thedriverless vehicle, and/or information to assist the driverless vehiclepast the area associated with the event. In some examples, thedriverless vehicle and the teleoperations system may be configured tocollaborate with one another, so that the driverless vehicle will beable to overcome the event. For example, upon encountering an event thedriverless vehicle may request guidance. Rather than simply instructingthe driverless vehicle how to navigate the event, the teleoperationssystem may provide guidance related to the request, and the driverlessvehicle may determine a course of operation to overcome the event basedat least in part on the guidance. In some examples, the driverlessvehicle may send a proposed maneuver and/or action to the teleoperatorand, based at least in part on the proposal, the teleoperator mayconfirm or reject the proposal.

In some examples, a method for operating a driverless vehicle includinga vehicle controller may include receiving, at the driverless vehicle,sensor signals including sensor data from one or more sensors associatedwith the driverless vehicle, wherein the sensor data is related tooperation of the driverless vehicle. The method may also includereceiving road network data from a road network data store, the roadnetwork data being based at least in part on a location of thedriverless vehicle. The method may further include determining, at thedriverless vehicle, a driving corridor within which the driverlessvehicle travels according to a trajectory. The driving corridor mayinclude virtual boundaries and may be based at least in part on thesensor data and/or the road network data. The method may also includecausing the driverless vehicle to traverse a road network autonomouslyaccording to a path from a first geographic location to a secondgeographic location different than the first geographic location. Themethod may also include determining that an event associated with thepath has occurred, and sending communication signals from the driverlessvehicle to a teleoperations system. The communication signals mayinclude a request for guidance from the teleoperations system and thesensor data, the road network data, or both, though any data and/oroutput from one or more modules of the vehicle systems is contemplated.The method may also include receiving, at the driverless vehicle,teleoperations signals from the teleoperations system. For example, theteleoperations signals may include guidance to alter the virtualboundaries of the driving corridor, such that the vehicle controllerdetermines a revised trajectory. For example, the alteration isconfigured to result in avoiding the event, traveling around the event,or passing through the event. In some examples, the teleoperationssignals may provide guidance to cause the vehicle controller to alterthe virtual boundaries a minimal amount that still allows the driverlessvehicle to traverse the event.

In some examples, the driverless vehicle may traverse the road networkautonomously by generating, at the driverless vehicle, a plurality ofrevised trajectories concurrently or substantially simultaneously(within technical capabilities) based at least in part on the alteredvirtual boundaries of the driving corridor. In some examples, each ofthe revised trajectories may be associated with a confidence level, andthe method may further include selecting a revised trajectory having ahighest confidence level from among the plurality of revisedtrajectories, and operating the driverless vehicle according to theselected revised trajectory. Confidence levels may be based at least inpart on a probability that the vehicle can traverse a portion of thepath. For example, confidence levels may be associated with aprobability that using a particular trajectory will result in thedriverless vehicle being able to successfully maneuver past a portion ofthe path associated with the event.

In some examples, an event may be identified in relation to a confidencelevel associated with a probability of the driverless vehicle being ableto successfully maneuver past a portion of the path between the firstgeographic location and the second geographic location. For example, ifthe confidence level is below a threshold confidence level, it may be anindication of the occurrence of an event that may result in initiationof transmission of communication signals from the driverless vehicle tothe teleoperations system including a request for guidance. In someexamples, the request may be inferred and/or determined by theteleoperations system based at least in part on, for example, the sensordata and/or other information associated with the driverless vehicle. Insome examples, determining that an event has occurred may includedetermining that a confidence level associated with a trajectoryaccording to which the vehicle is operating at a location along the pathis less than a threshold confidence level.

For example, an event may include one or more of an activity associatedwith a portion of the path, an object along the path at least partiallywithin the driving corridor as the vehicle approaches the object (e.g.,people, animals, vehicles, or other static or dynamic objects) along thepath at least partially within the driving corridor or moving with atrajectory toward the driving corridor as the vehicle approaches theobject. In some examples, the event may be indicative of a predictedmovement of an object into the driving corridor, resulting in aconfidence level dropping below the threshold confidence level. In someexamples, identifying an event may include determining a classificationof an object present in the driving corridor and/or predicting movementof an object having a trajectory toward the driving corridor. In suchevents, the confidence level associated with the vehicle successfullypassing the event according to its trajectory at the location associatedwith the event may be below a threshold confidence level. Suchcircumstances may result in initiation of transmission of communicationsignals from the driverless vehicle including a request for guidancefrom the teleoperations system.

This disclosure is also generally directed to a teleoperations systemfor assisting with operating a driverless vehicle. The driverlessvehicle may include a vehicle controller and may be configured toautonomously operate according to a first operating mode associated withfirst operating parameters via the vehicle controller along a roadnetwork according to a path from a first geographic location to adestination separated from the first geographic location. Theteleoperations system may include a teleoperations receiver configuredto receive sensor data associated with sensor signals received from onemore sensors associated with the driverless vehicle. The sensor data maybe related to operation of the driverless vehicle. The teleoperationssystem may also include a teleoperations interface configured tofacilitate determining that the driverless vehicle is in a secondgeographic area based at least in part on the sensor data associatedwith sensor signals received from one or more sensors associated withthe driverless vehicle. The teleoperations interface may also beconfigured to facilitate classifying the second geographic area ascorresponding to a zone in which the vehicle controller is to operatethe driverless vehicle according to a second operating mode associatedwith second operating parameters. One or more of the second operatingparameters may differ from a corresponding first operating parameter.The teleoperations system may also include a teleoperations transmitterconfigured to send teleoperations signals to the driverless vehicle. Theteleoperations signals may provide guidance to the vehicle controller toswitch from the first operating mode to the second operating mode whileoperating in the second geographic area.

This disclosure is also generally directed to a method for operating adriverless vehicle including a vehicle controller, and autonomouslyoperating the driverless vehicle according to a first operating modeassociated with first operating parameters via the vehicle controlleralong a road network according to a path from a first geographiclocation to a destination separated from the first geographic location.The method may include receiving, via a teleoperations receiver locatedremotely from the driverless vehicle, via another entity, and/or via thedriverless vehicle, communication signals indicating occurrence of anevent associated with a second geographic area located along the path.The method may additionally or alternatively include reviewing, by ateleoperator in communication with the teleoperations receiver, sensordata associated with sensor signals received from one more sensorsassociated with the driverless vehicle. In some examples, the sensordata may be related to operation of the driverless vehicle. The methodmay also include classifying, via the other entity and/or theteleoperator, the second geographic area as corresponding to a zone inwhich the vehicle controller operates the driverless vehicle accordingto a second operating mode associated with second operating parameters.In some examples, at least one of the second operating parameters maydiffer from a corresponding first operating parameter. The method mayfurther include sending teleoperations signals, via a teleoperationstransmitter, to the driverless vehicle. In some examples, theteleoperations signals may provide guidance to the vehicle controller toswitch from the first operating mode to the second operating mode whileoperating in the second geographic area. In some examples, theteleoperations signals may provide guidance representative of virtualboundaries of the second geographic area. The second operatingparameters may include one or more of second performance parameters,second vehicle operation policies, second vehicle operation laws, andsecond vehicle operation regulations. In some examples, the secondgeographic area may correspond to one or more of a construction zone, aschool zone, a flood zone, an accident zone, a parade zone, a specialevent zone, and a zone associated with a slow traffic condition. In someexamples, the teleoperations signals may provide guidance including oneor more of causing the driverless vehicle to at least one of ignore theevent, increase or decrease probabilities of classes of objects (e.g.,in a school zone, increase a probability that a small object is achild), alter virtual boundaries of a driving corridor within which thevehicle operates, and operate the driverless vehicle according to atravel speed constraint (e.g., reducing a maximum travel speed). Someexamples of the method may include sending teleoperations signals to aplurality of driverless vehicles. In some examples, the teleoperationssignals may include guidance to operate at least some of the pluralityof driverless vehicles in the second geographic area according thesecond operating mode.

This disclosure is also generally directed to a computer-readablestorage medium having computer-executable instructions stored thereuponwhich, when executed by a computer, cause the computer to assist withoperating at least a subset of a plurality of driverless vehicles of afleet of driverless vehicles. Each driverless vehicle of the subset mayinclude a vehicle controller and may autonomously operate according to afirst operating mode associated with first operating parameters via thevehicle controller along a road network according to a respective pathfrom a respective first geographic location to a respective destinationseparated from the first geographic location. Assisting with operatingthe at least a subset of the plurality of vehicles may cause thecomputer to send teleoperations signals to each driverless vehicle ofthe subset. The teleoperations signals may provide guidance to therespective vehicle controllers to switch from the first operating modeto a second operating mode. The second operating mode may be associatedwith second operating parameters including one or more of secondperformance parameters, second vehicle operation policies, secondvehicle operation laws, or second vehicle operation regulations. Atleast one of the second operating parameters may differ from acorresponding first operating parameter.

This disclosure is also generally directed to a method for alteringoperation of at least a subset of a plurality of driverless vehicles ofa fleet of driverless vehicles. Each driverless vehicle of the subsetmay include a vehicle controller and may autonomously operate accordingto a first operating mode associated with first operating parameters viathe vehicle controller along a road network according to a respectivepath from a respective first geographic location to a respectivedestination separated from the first geographic location. The method mayinclude sending teleoperations signals, via a teleoperations transmitterin communication with a teleoperator and located remotely from thedriverless vehicles, to each driverless vehicle of the subset. Theteleoperations signals may provide guidance to the respective vehiclecontrollers to switch from the first operating mode to a secondoperating mode. The second operating mode may be associated with secondoperating parameters including one or more of second performanceparameters, second vehicle operation policies, second vehicle operationlaws, and second vehicle operation regulations. At least one of thesecond operating parameters may differ from a corresponding firstoperating parameter. In some examples, the guidance may includeswitching from the first operating mode to the second operating mode fora predetermined period of time and thereafter returning to the firstoperating mode.

The second operating parameters may include one or more of reducingenergy expenditure of the driverless vehicles, setting a maximumoperating speed, preventing the driverless vehicles from operatingbidirectionally, changing a threshold confidence level required forautonomous operation, changing a threshold confidence level required forautonomous operation in a second geographic area, altering at least oneof an object classification model or an object prediction model used bythe driverless vehicles, or relaxing vehicle operation policiesassociated with complying with traffic laws and regulations.

In some examples, such teleoperations signals may also include commandsto actuate one or more systems or subsystems of at least a portion ofthe driverless vehicles in a fleet of driverless vehicles. As anon-limiting example, a second operating mode may be associated withturning on headlights, turning off interior lighting, controlling avolume of audio signals, relaying audio and/or visual light patterns toan interior or exterior of each driverless vehicle, turning on or offone or more sensors (e.g., LIDAR, RADAR, cameras, IMUs, etc.), or thelike.

In some examples, the teleoperations signals to each driverless vehicleof the subset may include guidance to the respective vehicle controllersto avoid a second geographic area based at least in part on a presenceof an event associated with the road network. For example, the secondgeographic area may correspond to one or more of a construction zone, aschool zone, a flood zone, an accident zone, a parade zone, a specialevent zone, or a zone associated with a slow traffic condition. In suchexamples, operating each of the driverless vehicles of the subset viathe respective vehicle controllers may include operating each of thedriverless vehicles according to a second operating mode thatcorresponds to at least one of the zones. In some examples, the guidancemay include switching from the first operating mode to the secondoperating mode while operating in a second geographic area.

The subset of driverless vehicles may include one or more of driverlessvehicles carrying at least one occupant, driverless vehicles having nooccupants, driverless vehicles including at least one battery having acharge below a threshold level of charge, and driverless vehiclesconfigured to determine a status of conditions associated with the roadnetwork. Vehicles having one or more of these different examplecharacteristics may be operated differently to account for, or takeadvantage of, the one or more characteristics. For example, a vehiclehaving a relatively low battery charge but no occupants, may take a pathhaving a shorter distance between a starting point and a destination toreduce battery use, even though a traffic condition along the shorterpath may result in a trip having a longer duration, which might be lessdesirable if the vehicle is occupied.

In some examples, the method may include receiving, via a teleoperationsreceiver located remotely from the driverless vehicles, communicationsignals from at least one of the driverless vehicles of the subsetindicating occurrence of an event. The communication signals indicatingoccurrence of the event may include communication signals indicative ofa classification of an object present in the driving corridor or movingwith a trajectory toward the driving corridor. In some such examples,sending teleoperations signals providing guidance to the vehiclecontroller to switch from the first operating mode to the secondoperating mode may include sending teleoperations signals providingguidance to at least one of alter the classification of the object orignore the object.

This disclosure is also generally directed to a teleoperations systemfor assisting with operating a driverless vehicle. The driverlessvehicle may include a vehicle controller and may be configured toautonomously operate via the vehicle controller along a road networkaccording to a path from a first geographic location to a destinationseparated from the first geographic location. The teleoperations systemmay include a teleoperations receiver configured to receivecommunication signals from the driverless vehicle. The communicationsignals may include at least a portion of sensor data from one or moresensors associated with the driverless vehicle. The at least a portionof sensor data may be related to operation of the driverless vehicle.The communication signals may also include data indicating occurrence ofan event associated with the path. The data indicating occurrence of theevent may include data indicating a confidence level associated with thepath is less than a threshold confidence level. The teleoperationssystem may further include a teleoperations interface configured tofacilitate reviewing the at least a portion of sensor data and the dataindicating occurrence of the event, and determining a level of guidanceto provide the driverless vehicle based at least in part on at least oneof the at least a portion of sensor data or the data indicatingoccurrence of the event. The teleoperations system may also include ateleoperations transmitter configured to transmit teleoperations signalsto the driverless vehicle. The teleoperations signals may includeguidance to operate the driverless vehicle according to the determinedlevel of guidance. The driverless vehicle may be configured to maneuvervia the vehicle controller to at least one of avoid the event, travelaround the event, or pass through the event based at least in part onthe teleoperations signals.

This disclosure is also generally directed to a method for operating adriverless vehicle. The driverless vehicle may include a vehiclecontroller and may autonomously operate via the vehicle controller alonga road network according to a path from a first geographic location to adestination separated from the first geographic location. The method mayinclude receiving, at a teleoperations receiver located remotely fromthe driverless vehicle, communication signals from the driverlessvehicle. The communication signals may include at least a portion ofsensor data from one or more sensors associated with the driverlessvehicle. The at least a portion of sensor data may be related tooperation of the driverless vehicle. The communication signals may alsoinclude data indicating occurrence of an event associated with the path.The data indicating occurrence of the event may include data indicatinga confidence level associated with the path is less than a thresholdconfidence level. The method may include reviewing, via a teleoperationssystem (e.g., via a teleoperator) in communication with theteleoperations receiver, the at least a portion of sensor data and thedata indicating occurrence of the event. The method may also includedetermining, via the teleoperations system, a level of guidance toprovide the driverless vehicle based at least in part on the at least aportion of sensor data and/or the data indicating occurrence of theevent. The method may also include transmitting teleoperations signals,via a teleoperations transmitter, to the driverless vehicle. In someexamples, the teleoperations signals may include guidance to operate thedriverless vehicle according to the determined level of guidance, sothat the vehicle controller maneuvers the driverless vehicle to avoidthe event, travel around the event, and/or pass through the event.

This disclosure is also generally directed to a method for operating adriverless vehicle. The driverless vehicle may include a vehiclecontroller and may autonomously operate via the vehicle controller alonga road network according to a path from a first geographic location to adestination separated from the first geographic location. The method mayinclude receiving, at a teleoperations receiver located remotely fromthe driverless vehicle, first communication signals from the driverlessvehicle. The first communication signals may include first sensor datafrom one or more sensors associated with the driverless vehicle. Thefirst sensor data may be related to operation of the driverless vehicle.The communication signals may also include data indicating occurrence ofa first event associated with the path. The first event may includefirst characteristics including one or more one characteristics notpreviously encountered by the driverless vehicle or one or morecharacteristics previously encountered by the driverless vehicle fewerthan a threshold number of occurrences. The first communication signalsmay also include a request for guidance to pass the event and continuealong the path. The method may include reviewing, via a teleoperationssystem in communication with the teleoperations receiver, dataassociated with the first communication signals received from thedriverless vehicle. The method may also include determining, via theteleoperations system, a first level of guidance for providing thedriverless vehicle based at least in part on the data associated withthe first communication signals. The method may also include sendingfirst teleoperations signals, via a teleoperations transmitter, to thedriverless vehicle. The first teleoperations signals may include thefirst level of guidance, so that the vehicle controller maneuvers thedriverless vehicle to pass the first event and continue along the pathaccording to the first level of guidance.

The method may further include receiving, via the teleoperationsreceiver, second communication signals from the driverless vehicle. Thesecond communication signals may include second sensor data from one ormore sensors associated with the driverless vehicle. The second sensordata may be related to operation of the driverless vehicle. The secondcommunication signals may also include data indicating occurrence of asecond event associated with the path. The second event may includesecond characteristics, and the second characteristics may include oneor more characteristics in common with one or more of the firstcharacteristics. The second communication signals may also include arequest for information related to the second event and/or a proposedaction for passing the second event and continuing along the path. Themethod my also include reviewing, via a teleoperations system incommunication with the teleoperations receiver, data associated with thesecond communication signals received from the driverless vehicle. Themethod may also include determining, via the teleoperations system, asecond level of guidance for providing the driverless vehicle based atleast in part on the data associated with the second communicationsignals. The method may also include sending second teleoperationssignals, via the teleoperations transmitter, to the driverless vehicle.In some examples, the second teleoperations signals may include thesecond level of guidance, and the second level of guidance may includethe information related to the second event and/or the proposed secondaction, so that the vehicle controller maneuvers the driverless vehicleto pass the second event and continue along the path based at least inpart on the information related to the second event and/or the proposedsecond action.

This disclosure is also generally directed to a method for operating aplurality of driverless vehicles. The driverless vehicles may eachinclude a vehicle controller and may autonomously operate via thevehicle controller along a road network according to a path from a firstgeographic location to a destination separated from the first geographiclocation. The method may include receiving, at a teleoperations receiverlocated remotely from a first one of the driverless vehicles, firstcommunication signals from the first driverless vehicle indicatingoccurrence of a first event associated with the road network along apath associated with the first driverless vehicle. The firstcommunication signals may include a request for guidance to pass theevent and continue along the path. The method may also includereviewing, by a teleoperator system in communication with theteleoperations receiver, data associated with the first communicationsignals received from the first driverless vehicle. The method may alsoinclude determining, by the teleoperations system, a first level ofguidance for providing the first driverless vehicle based at least inpart on the data associated with the first communication signals. Themethod may also include sending first teleoperations signals, via ateleoperations transmitter, to the first driverless vehicle. The firstteleoperations signals may include the first level of guidance, suchthat the vehicle controller maneuvers the first driverless vehicle topass the first event and continue along the path according to the firstlevel of guidance.

The method may also include receiving, at the teleoperations receiver,second communication signals from a second driverless vehicle of thedriverless vehicles indicating occurrence of a second event associatedwith the road network along a path associated with the second driverlessvehicle. The second event may include second characteristics, and thesecond characteristics may include at least one second characteristic incommon with one or more characteristics of the first event. The secondcommunication signals may include a request for information related tothe second event and/or a proposed action for passing the second eventand continuing along the path. The method may also include reviewing, bythe teleoperations system, data associated with the second communicationsignals received from the second driverless vehicle. The method may alsoinclude determining, by the teleoperations system, a second level ofguidance for providing the second driverless vehicle based at least inpart on data associated with the first event and the data associatedwith the second communication signals. The method may further includesending second teleoperations signals, via the teleoperationstransmitter, to the second driverless vehicle. The second teleoperationssignals may include the second level of guidance, and the second levelof guidance may include the information related to the second eventand/or the proposed second action, so that the vehicle controllermaneuvers the second driverless vehicle to pass the second event andcontinue along the path based at least in part on the informationrelated to the second event and/or the proposed second action.

The techniques and systems described herein may be implemented in anumber of ways. Example implementations are provided below withreference to the figures.

FIG. 1 is a schematic diagram of an example environment 100 throughwhich an example vehicle 102 travels. The example environment 100includes a road network 104 including a plurality of example roads 106having two pairs 108 of lanes 110 separated by a median or double-yellowline 112, with each of the lanes 110 of a pair 108 of lanes 110 definedby a lane dividing line 114 and lane boundary lines 116. The exampleroad 106 also includes shoulders 118 located on opposite sides of theroad 106. FIG. 1 also shows an example geographic location 120associated with a departure location including a structure 122, such asa house or building, and an example destination 124 also including astructure 126, such as a house or building. The road network 104provides a number of roads 106 defining a path between the geographiclocation 120 and the destination 124, and FIG. 1 shows an enlarged viewof a portion of an example road 106. The road network 104 may include anumber of features, such as curves, intersections with cross-roads,crosswalks, traffic signs, traffic lights, railroad crossings, trafficcircles, directional arrows, etc.

As shown in FIG. 1 , the example vehicle 102 may travel through theexample environment 100 via the road network 104 according to a pathfrom the geographic location 120 to the destination 124. For the purposeof illustration, the vehicle 102 may be a driverless vehicle, such as anautonomous vehicle configured to operate according to a Level 5classification issued by the U.S. National Highway Traffic SafetyAdministration, which describes a vehicle capable of performing allsafety-critical functions for the entire trip, with the driver (oroccupant) not being expected to control the vehicle at any time. In thatcase, since the vehicle 102 may be configured to control all functionsfrom start to completion of the trip, including all parking functions,it may not include a driver. This is merely an example, and the systemsand methods described herein may be incorporated into any ground-borne,airborne, or waterborne vehicle, including those ranging from vehiclesthat need to be manually controlled by a driver at all times, to thosethat are partially or fully autonomously controlled.

The example vehicle 102 shown in FIG. 1 is an automobile having fourwheels 128 and respective tires for each of the wheels 128. Other typesand configurations of vehicles are contemplated, such as, for example,vans, sport utility vehicles, cross-over vehicles, trucks, buses,agricultural vehicles, and construction vehicles. The vehicle 102 may bepowered by one or more internal combustion engines, one or more electricmotors, hydrogen power, any combination thereof, and/or any othersuitable power sources. In addition, although the example vehicle 102has four wheels 128, the systems and methods described herein may beincorporated into vehicles having fewer or a greater number of wheels,tires, and/or tracks. The example vehicle 102 has four-wheel steeringand may operate generally with equal performance characteristics in alldirections, for example, such that a first end 130 of the vehicle 102 isa front end of the vehicle 102 when travelling in a first direction 132,and such that the first end 130 becomes the rear end of the vehicle 102when traveling in the opposite, second direction 134, as shown in FIG. 1. Similarly, a second end 136 of the vehicle 102 is a front end of thevehicle 102 when travelling in the second direction 134, and such thatthe second end 136 becomes the rear end of the vehicle 102 whentraveling in the opposite, first direction 132. Such a configuration maybe referred to herein as “bidirectionality.” These example bidirectionalcharacteristics may facilitate greater maneuverability, for example, insmall spaces or crowded environments, such as parking lots and urbanareas.

In the example shown in FIG. 1 , the vehicle 102 may use various sensorsand a vehicle controller to autonomously operate through the environment100 along the path via the road network 104, as explained in more detailherein. For example, the vehicle controller may be configured todetermine a driving corridor 138 defined by virtual boundaries 140within which the vehicle 102 may travel. For example, the drivingcorridor 138 may have a variable corridor width 142 in the widthdirection of the vehicle 102, and a variable corridor length 144extending in the direction of travel of the vehicle 102. In someexamples, the virtual boundaries 140 of the driving corridor 138 may bedetermined based at least in part on sensor data received from sensorsassociated with the vehicle 102 and/or road network data received by thevehicle 102 via a road network data store, as explained in more detailherein. Though not illustrated in FIG. 1 , such sensor data indicativeof objects may be represented in such a corridor as indented or removedportions. In some examples, the vehicle 102 may travel along a driveline 146 within the driving corridor 138.

In some examples, the vehicle 102 may operate autonomously until thevehicle 102 encounters an event along the road 106 for which it mayrequest assistance from, for example, a teleoperations system 148located remotely from the vehicle 102. For example, the vehicle 102 mayencounter a construction zone associated with a portion of the path, andtraffic in the vicinity of the construction zone may be under thedirection of a construction worker who provides instructions for trafficto maneuver around the construction zone. Due in part to theunpredictable nature of this type of event, the vehicle 102 may requestremote assistance from the teleoperations system 148. In some examples,the vehicle 102 may be a part of a fleet of vehicles in communicationvia a communications network with the teleoperations system 148, asexplained in more detail herein.

In some examples, for example as shown in FIG. 1 , the teleoperationssystem 148 may include one or more teleoperators 150, which may be humanteleoperators located at a teleoperations center 152. In some examples,one or more of the teleoperators 150 may not be human. For example, theymay be computer systems leveraging artificial intelligence, machinelearning, and/or other decision making strategies. In the example shown,the teleoperator 150 may interact with one or more vehicles 102 in thefleet of vehicles via a teleoperator interface 154. The teleoperatorinterface 154 may include one or more displays 156 configured to providethe teleoperator 150 with data related to operation of the vehicle 102,a subset of the fleet of vehicles, and/or the fleet of vehicles. Forexample, the display(s) 156 may be configured to show data related tosensor signals received from the vehicles 102, data related to the roadnetwork 104, and/or additional data or information to facilitateproviding assistance to the vehicles 102. In addition, the teleoperatorinterface 154 may also include a teleoperator input device 158configured to allow the teleoperator 150 to provide information to oneor more of the vehicles 102, for example, in the form of teleoperationssignals providing guidance to the vehicles 102. The teleoperator inputdevice 158 may include one or more of a touch-sensitive screen, astylus, a mouse, a dial, a keypad, and/or a gesture-input systemconfigured to translate gestures performed by the teleoperator 150 intoinput commands for the teleoperator interface 154. As explained in moredetail herein, the teleoperations system 148 may provide one or more ofthe vehicles 102 with guidance to avoid, maneuver around, or passthrough events.

FIG. 2 is a block diagram of an example architecture 200 includingvehicle systems 202 for controlling operation of the systems thatprovide data associated with operation of the vehicle 102, and thatcontrol operation of the vehicle 102.

In various implementations, the architecture 200 may be implementedusing a uniprocessor system including one processor, or a multiprocessorsystem including several processors (e.g., two, four, eight, or anothersuitable number). The processor(s) may be any suitable processor capableof executing instructions. For example, in various implementations, theprocessor(s) may be general-purpose or embedded processors implementingany of a variety of instruction set architectures (ISAs), such as thex86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. Inmultiprocessor systems, each processor may commonly, but notnecessarily, implement the same ISA. In some examples, the processor(s)may include a central processing unit (CPU), a graphics processing unit(GPU), or a combination thereof.

The example architecture 200 may include a non-transitory computerreadable media configured to store executable instructions/modules,data, and/or data items accessible by the processor(s). In variousimplementations, the non-transitory computer readable media may beimplemented using any suitable memory technology, such as static randomaccess memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory. In theillustrated implementation, program instructions and data implementingdesired functions, such as those described above, are shown storedwithin the non-transitory computer readable memory. In otherimplementations, program instructions, and/or data may be received,sent, or stored on different types of computer-accessible media, such asnon-transitory media, or on similar media separate from thenon-transitory computer readable media. Generally speaking, anon-transitory, computer readable memory may include storage media ormemory media, such as flash memory (e.g., solid state memory), magneticor optical media (e.g., a disk) coupled to the architecture 200 via anI/O interface. Program instructions and data stored via a non-transitorycomputer readable medium may be transmitted by transmission media orsignals such as electrical, electromagnetic, or digital signals, whichmay be conveyed via a communication medium such as a network and/or awireless link, such as may be implemented via a network interface.

In some implementations, the I/O interface may be configured tocoordinate I/O traffic between the processor(s), the non-transitorycomputer readable media, and any peripheral devices, the networkinterface, or other peripheral interfaces, such as input/output devices.In some implementations, the I/O interface may perform any necessaryprotocol, timing, or other data transformations to convert data signalsfrom one component (e.g., the non-transitory computer readable media)into a format suitable for use by another component (e.g.,processor(s)). In some implementations, the I/O interface may includesupport for devices attached through various types of peripheral buses,such as a variant of the Peripheral Component Interconnect (PCI) busstandard or the Universal Serial Bus (USB) standard, for example. Insome implementations, the function of the I/O interface may be splitinto two or more separate components, such as a north bridge and a southbridge, for example. Also, in some implementations, some or all of thefunctionality of the I/O interface, such as an interface to thenon-transitory computer readable media, may be incorporated directlyinto the processor(s).

In the example architecture 200 shown in FIG. 2 , the example vehiclesystems 202 include a plurality of sensors 204, for example, configuredto sense movement of the vehicle 102 through the environment 100, senseenvironmental data, such as the ambient temperature, pressure, andhumidity, and/or sense objects in the environment 100 surrounding thevehicle 102. In some examples, the sensors 204 may include sensorsconfigured to identify a location on a map. The sensors 204 may include,for example, one or more light detection and ranging sensors (LIDAR),one or more cameras (e.g., RGB-cameras, intensity (grey scale) cameras,infrared cameras, depth cameras, stereo cameras, and the like), one ormore radio detection and ranging sensors (RADAR), one or more soundnavigation and ranging sensors (SONAR), one or more microphones forsensing sounds in the environment 100, such as sirens from lawenforcement and emergency vehicles, and other sensors related to theoperation of the vehicle 102. Other sensors may include a speed sensor,sensors related to operation of internal combustion engines and/orelectric motors, sensors related to the tires to detect tiretemperature, tire pressure, and tread depth, and/or brake-relatedsensors for detecting brake temperatures and/or wear, and in vehicleshaving regenerative braking, sensors for detecting parameters related tooperation of the regenerative braking system. The sensors 204 may alsoinclude, for example, inertial measurement units (IMUs), accelerometers,and gyroscopes. The sensors 204 may be configured to provide sensor data206 representative of the sensed objects and signals to the vehiclesystems 202 via, for example, an input/output (I/O) interface 208. Othertypes of sensors and sensor data are contemplated.

The example vehicle systems 202 also include location systems 210configured to receive location information, including position andorientation data (e.g., a local position or local pose) from the sensors204 and/or external sources, and provide location data 212 to otherportions of the vehicle systems 202 via the I/O interface 208. Theexternal sources may include global satellites for facilitatingoperation of a global positioning system (GPS) and/or a wireless networkfor communicating and receiving information related to the vehicle'slocation, such as map data. The location systems 210 may also includesensors configured to assist with navigation of the vehicle 102, such aswheel encoders for sensing the rotation of the wheels 128, inertialnavigation sensors, such as gyroscopes and/or accelerometers,magnetometers, and/or cameras for obtaining image data for visualodometry or visio-inertial navigation.

The example vehicle systems 202 also include one or more of a planner214, an object data calculator 216, an object classifier 218, acollision predictor system 220, a kinematics calculator 222, and asafety system actuator 224. The vehicle systems 202 may be configured toaccess one or more data stores including, but not limited to, an objecttype data store 226. The object type data store 226 may include datarepresenting object types associated with object classifications forobjects detected in the environment 100.

The example vehicle systems 202 shown in FIG. 2 also include a vehiclecontroller 228 configured to receive vehicle control data 230, and basedon the vehicle control data 230, communicate with a drive system 232(e.g., a steering system, a propulsion system, suspension system, and/ora braking system) to control operation of the vehicle 102. For example,the vehicle control data 230 may be derived from data received from oneof more of the sensors 204 and one or more of the planner 214, theobject data calculator 216, the object classifier 218, the collisionpredictor system 220, the kinematics calculator 222, and the safetysystem actuator 224, and control operation of the drive system 232, sothat operation and maneuvering of the vehicle 102 is executed.

In some examples, the planner 214 may be configured to generate datarepresentative of a trajectory of the vehicle 102, for example, usingdata representing a location of the vehicle 102 in the environment 100and other data, such as local pose data, that may be included in thelocation data 212. In some examples, the planner 214 may also beconfigured to determine projected trajectories predicted to be executedby the vehicle 102. The planner 214 may, in some examples, be configuredto calculate data associated with a predicted motion of an object in theenvironment 100, and may determine a predicted object path associatedwith the predicted motion of the object. In some examples, the objectpath may include the predicted object path. In some examples, the objectpath may include a predicted object trajectory. In some examples, theplanner 214 may be configured to predict more than a single predictedobject trajectory. For example, the planner 214 may be configured topredict multiple object trajectories based on, for example,probabilistic determinations or multi-modal distributions of predictedpositions, trajectories, and/or velocities associated with an object.

In some examples, the object data calculator 216 may be configured toprovide data representative of, for example, one or more of the pose(e.g., position and orientation) of an object in the environment 100surrounding the vehicle 102, an object track associated with the object(e.g., a historic position, velocity, acceleration, and/or heading ofthe object over a period of time (e.g., 5 seconds)), and an objectclassification associated with the object (e.g., a pedestrian, avehicle, a bicyclist, etc.). For example, the object data calculator 216may be configured to receive data in the form of sensor signals receivedfrom one or more of the sensors 204 and determine data representing oneor more of the position and/or orientation in the environment 100 of theobject, the object track, and the object classification.

In some examples, the object classifier 218 may be configured to accessdata from the object type data store 226, which may be configured tostore data representing object types, such as, for example, a species ofan object classification, a subclass of an object classification, and/ora subset of an object classification. The object classifier 218, in someexamples, may be configured to analyze data representing an object trackand data representing an object classification with data representing anobject type, and determine an object type based at least in part on theobject track and classification data. For example, a detected objecthaving an object classification of an “automobile” may have an objecttype of “sedan,” “coupe,” “hatch-back,” “sports utility vehicle,”“pick-up truck,” or “minivan.” An object type may include additionalsubclasses, designations, or subsets. For example, a “sedan” that isparked may have an additional subclass designation of being “static” or“being dynamic” if moving. In some examples, such an object classifiermay also determine a predicted object behavior based on one or more of aportion of the sensor data or the object type.

In some examples, the collision predictor system 220 may be configuredto use the data representing the object type, the predicted objectbehavior, the data representing the trajectory of the object, and/or thedata representing the trajectory of the vehicle 102, to predict acollision between the vehicle 102 and the object.

In some examples, the kinematics calculator 222 may be configured todetermine data representing one or more scalar and/or vector quantitiesassociated with motion of objects in the environment 100, including, butnot limited to, velocity, speed, acceleration, momentum, local pose,and/or force. Data from the kinematics calculator 222 may be used tocompute other data, including, but not limited to, data representing anestimated time to impact between an object and the vehicle 102, and datarepresenting a distance between the object and the vehicle 102. In someexamples, the planner 214 may use data produced by the kinematicscalculator 222 to estimate predicted object data. For example, theplanner 214 may use current scalar and/or vector quantities associatedwith object to determine a likelihood that other objects in theenvironment 100 (e.g., cars, motorcyclists, pedestrians, cyclists, andanimals) are moving in an alert or controlled state, versus an un-alertor uncontrolled state. For example, the kinematics calculator 222 may beconfigured estimate the probability that other objects are moving asthough they are being controlled and/or are behaving in a predictablemanner, or whether they are not being controlled and/or behaving in anunpredictable manner, for example, by observing motion of the objectover time and relative to other objects in the environment 100. Forexample, if the objects are moving erratically or without appearing toadjust to the presence or motion of other objects in the environment100, this may be an indication that the objects are either uncontrolledor moving in an unpredictable manner. This may be inferred based onsensor data received over time that may be used to estimate or predict afuture location of the object relative to a current or future trajectoryof the vehicle 102.

In some examples, the safety system actuator 224 may be configured toactivate one or more safety systems of the autonomous vehicle 102 when acollision is predicted by the collision predictor 220 and/or theoccurrence of other safety related events, such as, for example, anemergency maneuver by the vehicle 102, such as hard braking or a sharpacceleration. The safety system actuator 224 may be configured toactivate an interior safety system (e.g., including seat beltpre-tensioners and/or air bags), an exterior safety system (e.g.,including warning sounds and/or warning lights), the drive system 232configured to execute an emergency maneuver to avoid a collision, and/orany combination thereof. For example, the drive system 232 may receivedata for causing a steering system of the vehicle 102 to change thetravel direction of the vehicle 102, and a propulsion system of thevehicle 102 to change the speed of the vehicle 102 to alter thetrajectory of vehicle 102 from an initial trajectory to a trajectory foravoiding a collision.

The vehicle systems 202 may operate according to the following example.Data representing a trajectory of the vehicle 102 in the environment 100may be received by the vehicle controller 228. Object data associatedwith an object in the environment 100 surrounding the vehicle 102 may becalculated. Sensor data 206 from one or more of the sensors 204 may beused to calculate the object data. The object data may include datarepresenting the location of the object in the environment 100, anobject track associated with the object, such as whether the object isstationary or moving, and an object classification associated with theobject, such as whether the object is another vehicle, a pedestrian, acyclist, an animal, or a stationary object. In some examples, the objectdata calculator 216, based on the object data, may be used to determinedata representing the object's location in the environment 100, datarepresenting whether the object is moving, and data representing aclassification associated with the object.

In some examples, the planner 214 may use the object data to determine apredicted path of the object in the environment, for example, based ondata representing the location of the object and may process that datato generate data representing a predicted object path. Data representingthe type of object may be determined based on the data representingwhether the object is moving, data representing the object'sclassification, and/or data representing object's type. A pedestrian notin motion, a vehicle in motion, and traffic sign, a lane marker, or afire hydrant, none of which is in motion, are examples of object typeswith an associated motion data.

In some examples, the collision predictor system 220 may be used topredict a collision between the vehicle 102 and an object in theenvironment 100 based on the object type, whether the object is moving,the trajectory of the vehicle 102, the predicted path of the objectobtained from the planner 214. For example, a collision may be predictedbased in part on the object type due to the object moving, thetrajectory of the object being in potential conflict with the trajectoryof the vehicle 102, and the object having an object classification thatindicates the object is a likely collision threat. In some examples,such a collision prediction may also be based on a predicted objectbehavior. In some examples, each classification, or sub-classification,of objects may have a corresponding associated behavior. As anon-limiting example, a predicted behavior of a bicycle is to travel inrelatively straight lines having a maximum speed.

In some examples, the safety system actuator 224 may be configured toactuate one or more portions of a safety system of the vehicle 102 whena collision is predicted. For example, the safety system actuator 224may activate one or more of the interior safety systems, one or more ofthe exterior safety systems, and one or more of the components of thedrive system 232 (e.g., the steering system, the propulsion system,and/or the braking system) via the vehicle controller 228. In someexamples, the vehicle controller 228 may determine that the interiorsafety system will be activated based on some action of an object in theenvironment 100, and the vehicle control data 230 may includeinformation configured to cause the vehicle controller 228 to activateone or more functions of the interior safety system, the exterior safetysystem, and the drive system 232.

As shown in FIG. 2 , the example vehicle systems 202 also include anetwork interface 234 configured to provide a communication link betweenthe vehicle 102 and the teleoperations system 148. For example, thenetwork interface 234 may be configured to allow data to be exchangedbetween the vehicle 102, other devices coupled to a network, such asother computer systems, other vehicles 102 in the fleet of vehicles,and/or with the teleoperations system 148. For example, the networkinterface 234 may enable wireless communication between numerousvehicles and/or the teleoperations system 148. In variousimplementations, the network interface 234 may support communication viawireless general data networks, such as a Wi-Fi network. For example,the network interface 234 may support communication viatelecommunications networks, such as, for example, cellularcommunication networks, satellite networks, and the like.

In various implementations, the parameter values and other dataillustrated herein may be included in one or more data stores, and maybe combined with other information not described or may be partitioneddifferently into more, fewer, or different data structures. In someimplementations, data stores may be physically located in one memory ormay be distributed among two or more memories.

Those skilled in the art will appreciate that the example architecture200 is merely illustrative and is not intended to limit the scope of thepresent disclosure. In particular, the computing system and devices mayinclude any combination of hardware or software that can perform theindicated functions, including computers, network devices, internetappliances, tablet computers, PDAs, wireless phones, pagers, etc. Thearchitecture 200 may also be connected to other devices that are notillustrated, or instead may operate as a stand-alone system. Inaddition, the functionality provided by the illustrated components mayin some implementations be combined in fewer components or distributedin additional components. Similarly, in some implementations, thefunctionality of some of the illustrated components may not be providedand/or other additional functionality may be available.

Those skilled in the art will also appreciate that, while various itemsare illustrated as being stored in memory or storage while being used,these items or portions of them may be transferred between memory andother storage devices for purposes of memory management and dataintegrity. Alternatively, in other implementations, some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated architecture 200. Some or all of thesystem components or data structures may also be stored (e.g., asinstructions or structured data) on a non-transitory,computer-accessible medium or a portable article to be read by anappropriate drive, various examples of which are described above. Insome implementations, instructions stored on a computer-accessiblemedium separate from the architecture 200 may be transmitted to thearchitecture 200 via transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a wireless link. Various implementations may further includereceiving, sending, or storing instructions and/or data implemented inaccordance with the foregoing description on a computer-accessiblemedium. Accordingly, the techniques described herein may be practicedwith other control system configurations. Additional information aboutthe operations of the modules of the vehicle 102 is discussed below.

FIG. 3 shows an example architecture 300 including a fleet 302 and anexample teleoperations system 148. The example fleet 302 includes aplurality of vehicles 102, at least some which are communicativelycoupled to the teleoperations system 148, for example, via therespective network interfaces 234 of the vehicles 102, and ateleoperations receiver 304 and a teleoperations transmitter 306associated with the teleoperations system 148. For example, a vehicle102 may send communication signals via the network interface 234, whichare received by the teleoperations receiver 304. In some examples, thecommunication signals may include, for example, sensor data from sensorsignals generated by one or more sensors associated with the vehicle102, and/or road network data from a road network data store. In someexamples, the sensor data may include raw sensor data or processedsensor data, and the road network data may include data related to aglobal or local map of an area associated with operation of the vehicle102. In some examples, the communication signals may include dataassociated with the current status of the vehicle 102 and its systems,such as, for example, its current position, current speed, current pathand/or trajectory, current occupancy, the level of charge of one or moreof its batteries, and/or the operational status of its sensors and drivesystems. In some examples, the communication signals from the vehicle102 may include a request for information from the teleoperations system148. Such information, may include, for example, assistance withoperation of the vehicle 148 in the form of, for example, informationabout objects, the road network 104, the road 106, the global map, thelocal map, collaboration with respect to vehicle operations andmaneuvers, and/or confirmation of information and/or actions proposed bythe vehicle 102.

As shown in FIG. 3 , the teleoperations receiver 304 may becommunicatively coupled to the teleoperations interface 154, and in someexamples, the teleoperator 150 may be able to access the sensor data,the road network data, and/or any other data in the communicationsignals received from a vehicle 102 via the teleoperations interface154. In some examples, the teleoperator 150 may be able to selectivelyaccess the sensor data, road network data, and/or other data via theinput device 158 and view the selected data via one or more of thedisplays 156 (see FIGS. 1 and 4 ). In some examples, the displays 156may display simplistic pictorial representations, animations, boundingboxes, arrows indicating a bearing and/or velocity of objects, iconsrepresenting objects, colorization of the sensor data, and/or otherrepresentations of the data, which may simplify interpretation by ateleoperator 150.

In the example shown, the teleoperations system 148 also includes ateleoperations network 308 configured to provide communication betweentwo or more of the teleoperations interfaces 154 and the respectiveteleoperators 150, and/or communication with teleoperations data 310.For example, the teleoperations system 148 may include a plurality ofteleoperations interfaces 154 and respective teleoperators 150, and theteleoperators 150 may communicate with one another via theteleoperations network 308 to facilitate and/or coordinate the guidanceprovided to the vehicles 102 of the fleet 302. In some examples, theremay be a teleoperator 150 assigned to each of the vehicles 102, and insome examples, a teleoperator 150 may be assigned to more than a singlevehicle 102 of the fleet 302. In some examples, more than oneteleoperator 150 may be assigned to a single vehicle 102. In someexamples, teleoperators 150 may not be assigned to specific vehicles 102of the fleet 302, but may instead provide guidance to vehicles 102 thathave encountered certain types of events and/or to vehicles 102 basedon, for example, a level of urgency associated with the vehicle'sencounter with the event. In some examples, data associated with anevent and/or the guidance provided by a teleoperator 150 may be storedby the teleoperations system 148, for example, in storage for theteleoperations data 310, and/or accessed by other teleoperators 150.

In some examples, the teleoperation data 310 may be accessible by theteleoperators 150, for example, via the teleoperations interface 154,for use in providing guidance to the vehicles 102. For example, theteleoperations data 310 may include global and/or local map data relatedto the road network 104, events associated with the road network 104,and/or travel conditions associated with the road network 104 due to,for example, traffic volume, weather conditions, construction zones,and/or special events. In some examples, the teleoperations data 310 mayinclude data associated with one more of the vehicles 102 of the fleet302, such as, for example, maintenance and service information, and/oroperational history including, for example, event history associatedwith the vehicle 102, path histories, occupancy histories, and othertypes of data associated with the vehicle 102.

FIG. 4 shows an example teleoperations interface 154. The example shownincludes three displays 156A, 156B, and 156C configured to provide theteleoperator 150 with data related to operation of one or more vehicles102 of the fleet 302. For example, the displays 156A, 156B, and 156C maybe configured to show data related to communication signals receivedfrom the vehicles 102, data related to the road network 104, and/oradditional data or information to facilitate providing assistance to thevehicles 102. In some examples, the different displays 156 may beconfigured to show different information related to one or more of thevehicles 102. In some examples, such content may be presented asanimations or pictorial representations, colorization of sensor data,abstractions of sensor data (e.g., bounding boxes), or the like, so thatinformation may be readily apparent regarding an event.

For example, the display 156A may be configured to show an overhead mapview 400 of a geographic area in which one or more of the vehicles 102is travelling. In some examples, the map view 400 may be supplementedwith information related to the vehicle 102 and/or operationalconditions in the geographic area shown in the map view 400. In someexamples, the map view 400 may take the form of a split-screen view, forexample, with one portion of the display 156A showing the overhead mapview and another portion of the display 156A showing, for example,status-related data for a subset of the fleet 302 being monitored by theteleoperator 150 associated with the teleoperator interface 154. Othertypes of views are contemplated.

In some examples, the display 156B may be configured to show a situationview 402 that depicts, for example, a view from the perspective of thevehicle 102. Such as view may provide the teleoperator 150 with arelatively more intuitive view of a situation or event being experiencedby the vehicle 102. In some examples, the situation view 402 may takethe form of one or more of a live (real-time) video feed and a livesensor view providing a depiction of objects and surroundings sensed bythe sensors of the vehicle 102. In some examples, the sensor view mayprovide the teleoperator 150 with information related to whether thesensors are detecting all of the relevant objects in the surroundings.In some examples, the situation view 402 may take the form of asplit-screen view with one portion of the display 156B showing a livevideo feed and another portion of the display 156B showing a live sensorview. Other types of views and/or representations (e.g., such as thosedescribed herein) are contemplated.

The display 156C may be configured to show a fleet view 404 thatdepicts, for example, an overhead map view showing the locations of oneor more vehicles 102 of the fleet 302 and other information related tothe status of the fleet 302 and/or the geographic area shown, such as,for example, traffic-related information and/or event-relatedinformation. In some examples, the fleet view 404 may show the origins,destinations, and/or paths for one or more of the vehicles 102 of thefleet 302. In some examples, the fleet view 404 may take the form of asplit-screen view, for example, with one portion of the display 156Cshowing the overhead map view and another portion of the display 156Cshowing, for example, status-related data for a subset of the fleet 302being monitored by the teleoperator 150 associated with the teleoperatorinterface 154.

Although the displays 156A, 156B, and 156C are described in a mannerthat suggests they may be three separate displays 156, they may beintegrated into a single display 156, or may include fewer or moredisplays 156. In some examples, the displays 156 may be reconfigurableto show different information, information in different formats,information in different arrangements, and/or information at a differentlevel of detail. For example, the information displayed and/or thearrangement of the information displayed may be tailored by theteleoperator 150 associated with the teleoperator interface 154. In someexamples, the teleoperations system 148 may be configured toautomatically show the displayed information according to defaultsettings that provide, for example, the most useful information in themost intuitive arrangement and/or level of detail based on, for example,the situation and/or status associated with a vehicle 102 for whichguidance from the teleoperator 150 is most urgently needed.

In addition, the example teleoperator interface 154 shown in FIG. 4 alsoincludes a teleoperator input device 158 configured to allow theteleoperator 150 to provide information to one or more of the vehicles102, for example, in the form of teleoperations signals providingguidance to the vehicles 102. The teleoperator input device 158 mayinclude one or more of a touch-sensitive screen, a stylus, a mouse, adial, a keypad, and/or a gesture-input system configured to translategestures performed by the teleoperator 150 into input commands for theteleoperator interface 154. In some examples, the input device 158 mayinclude a split-screen providing different touch-sensitive areas thatthe teleoperator 150 may use to provide different types of informationto a vehicle 102. For example, the different areas of the split-screenmay provide the teleoperator 150 with different types of information andmay facilitate the teleoperator's ability to provide instructions to thevehicle 102, collaborate with the vehicle 102, and/or confirminformation and/or proposed actions to be executed by the vehicle 102.For example, one portion of the input device 158 may provide a menu ofdifferent vehicles 102 of the fleet 302 to facilitate the teleoperator'sselection of a vehicle 102 for which to provide guidance. Other portionsof the input device 158 may include interactive displays and/or optionsfor providing guidance to a selected vehicle 102, as explained in moredetail herein.

FIG. 5A is an illustrative user interface (UI) 500A to facilitateinteraction between a teleoperator 150 and an example vehicle 102 in afirst example event scenario in which an example static object is in theroad. The example UI 500A may be displayed via one or more of thedisplays 156A, 156B, and 156C. The example UI 500A shown in FIG. 5Aincludes a vehicle selection zone 502 providing multiple selectors 504for selecting one of a number of vehicles 102 of the fleet 302 aboutwhich to display information related to its operation. In the exampleshown, the vehicle selection zone 502 includes selectors 504A, 504B,504C, 504D, and 504E for selecting one of the vehicles designated,respectively, AV 001, AV 002, AV 003, AV 004, or AV 005. As shown inFIG. 5A, the information displayed relates to AV 001 as indicated by theshading of selector 504A, which corresponds to AV 001.

In some examples, such as shown, the selectors 504 each include statussnapshots 506A, 506B, 506C, 506D, and 506E providing information relatedto the status of the respective vehicles 102. For example, statussnapshots 506A and 506B indicate that AV 001 and AV 002 are “En Route,”indicating they are currently between a start point and destination ofrespective trips. The status snapshots 506C and 506D indicate that AV003 and AV 004 are “Roaming,” indicating they are traveling the roadnetwork 104 without a defined destination. The example status snapshot506E indicates that AV 005 is currently “Mapping,” indicating that AV005 is currently collecting data related to local map data and/or globalmap data. In some examples, the status snapshots 506 may also include acharge indicator 508, an occupancy indicator 510, and a vehicle speedindicator 512. The charge indicator 508 may show the level of chargeremaining in one or more batteries of the vehicle 102. The exampleoccupancy indicator 510 may show that the vehicle 102 has four seats,depicted as squares, with one seat occupied, as depicted by the shadedsquare. Some examples of the status snapshot 506 may also include a tripstatus bar 514 providing an indication of the amount of a planned tripof the vehicle 102 that has been completed. The form of the icons shownin FIG. 5A are exemplary, and icons having other forms are contemplated.The information shown in the status snapshots 506 is exemplary, andadditional or other information may be shown.

The example UI 500A shown in FIG. 5A also includes a viewer selector bar516, which includes view selector icons 518A, 518B, 518C, and 518Dpermitting the teleoperator to select a type of view shown in the UI500A. In the example shown, the view selector icon 518A, if selected,may provide an interface for the teleoperator 150 to set preferences(e.g., default preferences) for the information shown by the UI 500A.The view selector icon 518B, selected in FIG. 5A as indicated by theshaded box 520 surrounding the view selector icon 518B, provides theteleoperator 150 with a view of the selected vehicle 102, for example,as shown in FIG. 5A. The view selector icon 518C, if selected, mayprovide the teleoperator 150 with a view of a map showing the roadnetwork 104 in an area relevant to operation of one or more of thevehicles 102 of the fleet 302. The view selector icon 518D, if selected,may facilitate configuring the information and/or the arrangement ofinformation shown by the UI 500A. Although a three-dimensionalrepresentation of the selected vehicle 102 is illustrated in FIGS.6A-9C, in some examples, any permutation of sensor data, operation statedata, and/or teleoperations data may be presented in the UI, whether bya default, a setting, and/or a teleoperator input (e.g., via selectionof a view selector icon).

For example, as shown in FIG. 5A, the view selector icon 518B has beenselected, and the UI 500A includes an active view zone 522 providing areal-time simulated (or animated) perspective view of the vehicle 102selected via the selector 504A. In the example shown, the active viewzone 522 shows an animation depicting the vehicle 102 encountering anobject 524 in the road 106. The teleoperator 150 may use the active viewzone 522 to monitor the operation of, and the teleoperator's interactionwith, the selected vehicle 102 (i.e., AV 001 in this example) before,during, and/or after the teleoperator 150 interacts with the vehicle 102via the teleoperator interface 154. For example, the vehicle 102 maysend communications signals to the teleoperator system 148 includingsensor signals from one or more sensors associated with the vehicle 102and/or a request for guidance and/or information from the teleoperationssystem 148. Based at least in part on the communications signals, theactive view zone 522 provides a real-time perspective view of thevehicle 102 and the relevant environment. In some examples, the activeview zone 522 may display any permutation of sensor data, operationstate data, and/or teleoperations data. In some examples, as discussedherein, the permutation displayed may be determined a default, asetting, and/or a teleoperator input.

The example UI 500A shown in FIG. 5A also includes an overhead view zone526, which provides an animated overhead view corresponding to the viewshown in the active view zone 522. This provides the teleoperator 150with an alternative view, which may assist the teleoperator 150 insituations for which an overhead view may facilitate the teleoperator'sinteraction with the vehicle 102.

The example UI 500A shown in FIG. 5A also includes a video view zone528. In some examples, the video view zone 528 may provide a real-timevideo view from a video camera associated with the vehicle 102. In someexamples, any data discussed herein as being “real-time” mayadditionally or alternatively include real-time data and/or historicaldata. This may assist the teleoperator 150 with quickly understandingthe situation encountered by the vehicle 102. The example video viewzone 528 also includes a view selector 530 which facilitates theteleoperator's selection of one of the video cameras associated with thevehicle 102. For example, the vehicle 102 may include cameras providingviews from the perspective of one or more of the front, the sides, andthe rear of the vehicle 102, and the view selector 530 may permitselection of one of the cameras from which to provide real-time video.In the example shown, a selector control 532 corresponding to the frontcamera has been selected, and the video view zone 528 shows the viewfrom the front video camera of the vehicle 102.

The example UI 500A shown in FIG. 5A also includes an interaction bar534, which may provide an indication of how the teleoperator 150 isinteracting with a vehicle 102. For example, the interaction bar 534 mayinclude icons for indicating whether the teleoperator 150 is interactingwith the vehicle 102, which may be depicted by highlighting of the“Engaged” icon. If the teleoperator 150 is communicating with thevehicle 102 by identifying an area the vehicle 102 should avoid, the“Area-Avoid” icon may be highlighted. If the teleoperator 150 iscommunicating with the vehicle 102 by changing the driving corridor 138,the “Drive Corridor” icon may be highlighted. Other additional oralternate icons and/or related information are contemplated.

The example UI 500A shown in FIG. 5A also includes an alert bar 536configured to provide an indication of the confidence level of thevehicle 102 (e.g., as determined by the vehicle systems 202) and/or anevent being encountered by the vehicle 102. For example, as shown inFIG. 5A, the alert bar 536 indicates that the vehicle 102 has aconfidence level of 35% as denoted by the alert bar 536 displaying“System Confidence 35%,” and that the vehicle 102 has encountered astatic object 524, and that the vehicle 102 has a confidence level of100% that the object 524 is in the driving corridor 138 and that it is astatic object. Other forms and types of alerts are contemplated. In someexamples, a vehicle confidence level below a threshold confidence level(e.g., 90%, 85%, or 75%) may result in the vehicle 102 notifying theteleoperations system 148 of the status of the vehicle 102 by sendingcommunication signals to the teleoperations system 148. Other thresholdconfidence levels are contemplated. In some examples, one or more of theactive view zone 522, the overhead view zone 526, or the video view zone528 may include an alert icon 538, which may include an exclamationpoint, for example, as shown.

In some examples, the UI 500A may include color-coded information. Forexample, alerts may be depicted in red, the driving corridor 138 may bedepicted in green, and objects may be depicted in pink or red. Othercolor-coded combinations are contemplated.

FIG. 5A shows an example vehicle 102 in a first example event scenarioin which the example static object 524 is in the road 106. In someexamples, as the vehicle 102 approaches the object 524, the sensors 204(FIG. 2 ) associated with the vehicle 102 may detect the object 524.Once detected, one or more of the planner 214, the object datacalculator 216, the object classifier 218, the collision predictorsystem 220, and the kinematics calculator 222 (FIG. 2 ) may be used todetermine the location of the object 524, classify the object 524,determine whether the object 524 is static or dynamic, and if the objectis dynamic, predict a possible trajectory of the object 524. As thevehicle 102 approaches the object 524, one or more of these systems maybe used to calculate a confidence level associated with a probabilitythat the vehicle 102 will be able to successfully maneuver past theobject 524, for example, without assistance from the teleoperationssystem 148. As the confidence level drops below a threshold minimumconfidence level, the vehicle 102 may slow its speed or stop, and useits network interface 234 (FIG. 2 ) to send communication signals to theteleoperations system 148 providing sensor data and a request forguidance from the teleoperations system 148. In some examples, therequest may be inferred and/or determined by the teleoperations system148 based at least in part on, for example, the sensor data and or otherinformation associated with the vehicle 102, such as its change inspeed, confidence level, and/or other maneuvering that might beindicative of a need for guidance from the teleoperations systems 148.In some examples, the request may be inferred and/or determined based atleast in part on the location of the vehicle 102 and/or knowledge of anevent occurring at the location. The teleoperations receiver 304 (FIG. 3) of the teleoperations system 148 may receive the communicationsignals, the situation may be evaluated by a teleoperator 150 via theteleoperations interface 154 (with or without accessing theteleoperations data 310 via the teleoperations network 308), and theteleoperator 150 may send teleoperations signals to the vehicle 102 viathe teleoperations transmitter 306 providing guidance, for example, asdescribed herein.

In the example shown in FIG. 5A, the vehicle 102 approaches the object524, for example, a garbage can on its side, until the driving corridor138 overlaps with the object 524, at which point the confidence leveldrops below the threshold confidence level. The vehicle 102 sendscommunication signals to the teleoperations system 148 including arequest for guidance, for example, as described herein. In the exampleshown, alert icons 538 are displayed in the active view zone 522 and theoverhead view zone of the UI 500A to draw the teleoperator's attentionto the object 524. The teleoperator 150 may use the teleoperatorinterface 154 to provide the requested guidance.

FIG. 5B is an example UI 500B configured to facilitate interactionbetween the teleoperator 150 and the example vehicle 102 in the exampleevent scenario shown in FIG. 5A. In the example shown, the teleoperator150 has selected the view selector icon 518B, so the UI 500B displaysinformation related to the vehicle 102. For example, the active viewzone 522 shows an animated perspective depiction of the vehicle 102 asit approaches the object 524. The overhead view zone 526 shows ananimated overhead view of the vehicle 102 as it approaches the object524, and the video view zone 528 shows a live video camera view of theobject 524 in front of the vehicle 102. In some examples, theteleoperator 150 may select live video feed from a different camera byselecting a different one of the buttons 532 of the view selector 530.

The UI 500B shown in FIG. 5B shows example teleoperator 150 interactionwith the vehicle 102. The example interaction bar 534 indicates that theteleoperator 150 is engaged with the vehicle 102, depicted by the shaded“Engaged” icon, and that the teleoperator 150 is altering the boundaries140 of the driving corridor 138, depicted by the “Drive Corridor” iconbeing shaded. For example, the teleoperator 150 has expanded theboundaries 140 of the driving corridor 138 to the second lane 110,across the lane dividing line 114. In some examples, the teleoperator150 may accomplish this using the teleoperations input device 158 (FIG.4 ), which may involve the use of one or more of a touch-sensitivescreen, a stylus, a mouse, a dial, a keypad, and/or a gesture-inputsystem. Based on the teleoperator's 150 inputs, the teleoperationssystem 148 may transmit teleoperations signals to the vehicle 102 viathe teleoperations transmitter 306 (FIG. 3 ). In the example shown, thevehicle 102 may expand the boundaries 140 of its driving corridor 138 ina manner consistent with the teleoperations signals, for example, asshown in FIG. 5B. Upon expansion of the driving corridor 138, thevehicle 102 may generate, for example, via the vehicle systems 202 (FIG.2 ), a plurality of revised trajectories (e.g., concurrently orsubstantially simultaneously within technical capabilities) based atleast in part on the altered boundaries 140 of the driving corridor 138.In the example shown, the alert bar 536 displays a revised confidencelevel (“System Confidence 95%) that is above the threshold confidencelevel. In some examples, the vehicle 102 may calculate a confidencelevel for each of the revised trajectories, and the vehicle 102 mayselect a revised trajectory having the highest confidence level fromamong the plurality of revised trajectories. Based at least in part onthe selected revised trajectory, the vehicle 102 may determine a reviseddrive line 146 for use in maneuvering around the object 524. Thereafter,the vehicle controller 228 (FIG. 2 ) may be configured to operate thevehicle 102 according to the revised drive line 146, for example, asshown in FIG. 5B, and maneuver around the object 524.

In some examples, the communication signals from the vehicle 102 mayinclude a proposed drive line 146 to maneuver around the object 524, andthe teleoperations system 148 may evaluate the proposed drive line 146and determine that the proposed revised drive line 146 is eitheracceptable or unacceptable. If acceptable, the teleoperations system 148may send teleoperations signals including an indication of approval ofthe revised drive line 146 proposed by the vehicle 102. If unacceptable,the teleoperations system 148 may determine an alternative proposeddrive line, and send teleoperations signals to the vehicle 102,including a denial of the revised drive line 146 proposed by the vehicle102 and the alternative proposed drive line for evaluation and possibleexecution by the vehicle 102.

In some examples, the teleoperations interface 154 may be configured topermit the teleoperator 150 to advise the teleoperations system 148and/or other vehicles 102 of the fleet 302 about the object 524 in theroad 106. For example, the teleoperations interface 154 may facilitateidentification of the location and information associated with theobject 524 (e.g., its classification and/or whether it is static ordynamic) for use by the teleoperations system 148 and/or other vehicles102 of the fleet 302. This information may result in vehicles 102avoiding the area associated with the object 524 or may provide guidancefor vehicles 102 that encounter the object 524 and/or teleoperatorsassisting vehicles 102 as they encounter the object 524. In someexamples, as time passes from the initial encounter with the object 524,the teleoperations system 148 may reduce the confidence level associatedwith the information related to the object 524, for example, untilanother vehicle 102 confirms that the object 524 remains in in the road106 or that the object 524 is no longer in the road 106.

FIG. 6A is an example UI 600A to showing an example vehicle 102 in asecond example event scenario in which an example dynamic object 602 isin the road 106. In the example shown, the dynamic object 602 is a dog.As the vehicle 102 approaches the dynamic object 602, the sensors 204associated with the vehicle 102 may detect the object 602, and oncedetected, the vehicle 102 may determine the location of the dynamicobject 602, classify the dynamic object 602, and/or predict a possibletrajectory of the dynamic object 602. As the vehicle 102 approaches thedynamic object 602, the vehicle 102 may calculate a confidence levelassociated with a probability that the vehicle 102 will be able tosuccessfully maneuver past the dynamic object 602, for example, withoutassistance from the teleoperations system 148. As the confidence leveldrops below a threshold minimum confidence level, the vehicle 102 mayslow its speed or stop, and use its network interface 234 to sendcommunication signals to the teleoperations system 148 providing sensordata and a request for guidance from the teleoperations system 148. Insome examples, the request may be inferred and/or determined by theteleoperations system 148, for example, as noted above. Theteleoperations system 148 may receive the communication signals, thesituation may be evaluated by a teleoperator 150 via the teleoperationsinterface 154, and the teleoperator 150 may send teleoperations signalsto the vehicle 102 via the teleoperations transmitter 306 providingguidance, for example, as described herein.

In some examples, the alert bar 536 may show that a dynamic object 602has been detected, and the confidence level associated with thisdetection, for example, as shown in FIG. 6A. The UI 600A may alsoprovide an indication that the dynamic object 602 is dynamic byincluding one or more arrows 604 in one or more of the views of thescenario. In some examples, arrows 604 may provide an indication of apredicted trajectory of the dynamic object 602 based on, for example,its type and/or classification, and/or its prior and/or currenttrajectory.

FIG. 6B is an example UI 600B configured to facilitate interactionbetween the teleoperator 150 and the example vehicle 102 in the exampleevent scenario shown in FIG. 6A. In the example shown, the exampleinteraction bar 534 indicates that the teleoperator 150 is engaged withthe vehicle 102, depicted by the shaded “Engaged” icon, and that theteleoperator 150 is altering the boundaries 140 of the driving corridor138, depicted by the “Drive Corridor” icon being shaded. For example,the teleoperator 150 has expanded the boundaries 140 of the drivingcorridor 138 to the second lane 110, across the lane dividing line 114.In some examples, the teleoperator 150 may accomplish this using theteleoperations input device 158. Based on the teleoperator's 150 inputs,the teleoperations system 148 may transmit teleoperations signals to thevehicle 102 via the teleoperations transmitter 306. In the exampleshown, the vehicle 102 may expand the boundaries 140 of its drivingcorridor 138 in a manner consistent with the teleoperations signals, forexample, as shown in FIG. 6B. Upon expansion of the driving corridor138, the vehicle 102 may generate a plurality of revised trajectoriesbased at least in part on the altered boundaries 140 of the drivingcorridor 138. The vehicle 102 may calculate a confidence level for eachof the revised trajectories, and the vehicle 102 may select a revisedtrajectory having the highest confidence level from among the pluralityof revised trajectories. Based at least in part on the selected revisedtrajectory, the vehicle 102 may determine a revised drive line 146 foruse in maneuvering around the dynamic object 602. Thereafter, thevehicle controller 228 may be configured to operate the vehicle 102according to the revised drive line 146, for example, as shown in FIG.6B, and maneuver around the dynamic object 602.

In some examples, however, the teleoperator 150 may determine adifferent course of action. For example, the dynamic object 602 shown inFIGS. 6A and 6B is an unattended animal (e.g., a dog not on a leash andwithout movement being controlled by a person). The future movement ofan unattended animal may be particularly unpredictable, and thus,expanding the boundaries 140 of the driving corridor 138, for example,as shown in FIG. 6B may not be acceptable. For example, expanding theboundaries 140 of the driving corridor 138 may not result in increasingthe confidence level above the threshold confidence level. Thus, in somesuch examples, the teleoperator 150 may send teleoperations signalsproviding guidance for the vehicle 102 to encourage the animal to removeitself from the road 106. For example, the guidance may take the form ofa proposal for the vehicle controller 228 to move the vehicle 102 slowlyforward, for example, a predetermined distance and stop. Such movementmay encourage the animal to exit the road 106. Alternatively, or inaddition, the guidance may include a proposal for the vehicle 102 tosound an audible warning and/or activate lighting to provide a visualwarning. Some vehicles 102 may include speakers (or other forms of noisegenerator) and lighting, and the vehicle 102 may activate the speakersand/or the lighting, which may encourage the animal to exit the road106, after which, the vehicle 102 may return to its original drive line146 or a similar trajectory, depending on whether the animal has exitedthe road 106.

FIG. 7A is an example UI 700A showing a third event scenario in whichthe vehicle 102 has encountered an example construction zone 702 thatincludes both example static objects 704 and an example dynamic object706 in the road 106. In the example shown, the static objects 704 aretraffic cones, and the dynamic object 706 is a person. As the vehicle102 approaches the construction zone 702, the sensors 204 associatedwith the vehicle 102 may detect the static objects 704 and/or thedynamic object 706, and once detected, the vehicle 102 may determine thelocation of the static objects 704 and the dynamic object 706, classifythem, and/or predict a possible trajectory of the dynamic object 706. Asthe vehicle 102 approaches the construction zone 702, the vehicle 102may calculate a confidence level associated with a probability that thevehicle 102 will be able to successfully maneuver past the constructionszone, for example, without assistance from the teleoperations system148. As the confidence level drops below a threshold minimum confidencelevel, the vehicle 102 may slow its speed or stop, and use its networkinterface 234 to send communication signals to the teleoperations system148 providing sensor data and a request for guidance from theteleoperations system 148. In some examples, the request may be inferredand/or determined by the teleoperations system 148. The teleoperationssystem 148 may receive the communication signals, the situation may beevaluated by a teleoperator 150 via the teleoperations interface 154,and the teleoperator 150 may send teleoperations signals to the vehicle102 via the teleoperations transmitter 306 providing guidance.

In the example shown in FIG. 7A, the alert bar 536 of the UI 700Aindicates for the teleoperator 150 that the vehicle 102 has arrived at aconstruction zone, and that there is at least one static object and atleast one dynamic object associated with the construction zone, andfurther, that the confidence level associated with each of thoseidentifications is 100%. The alert bar 536 also indicates that theconfidence level of the vehicle 102 is 35% by displaying “SystemConfidence 35%.” In the example shown, one or more arrows 708 mayprovide an indication of a predicted trajectory of the dynamic object706 based on, for example, its type and/or classification, and/or itsprior and/or current trajectory.

FIG. 7B is an example UI 700B configured to facilitate interactionbetween the teleoperator 150 and the example vehicle 102 in order toovercome the example construction zone 702 shown in FIG. 7A. In theexample shown, a second vehicle 710 is approaching the vehicle 102 inthe second lane 110, thus rendering it dangerous to cross the dividingline 114, which, in this example, is a double-yellow line, to maneuveraround the construction zone 702. The example interaction bar 534indicates that the teleoperator 150 is engaged with the vehicle 102,depicted by the shaded “Engaged” icon, and that the teleoperator 150 isaltering the boundaries 140 of the driving corridor 138, depicted by the“Drive Corridor” icon being shaded. As shown, the teleoperator 150 hasexpanded the boundaries 140 of the driving corridor 138 to the shoulder712 of the road 106 on the same side of the dividing line 114 as thevehicle 102 to avoid the oncoming second vehicle 710 and permitting thevehicle 102 to maneuver past the construction zone 702, which is notpresent in the shoulder 712. In some examples, the teleoperator 150 mayaccomplish this using the teleoperations input device 158, for example,as noted above. Based on the teleoperator's inputs, the teleoperationssystem 148 may transmit teleoperations signals to the vehicle 102 viathe teleoperations transmitter 306. In the example shown, the vehicle102 has expanded the boundaries 140 of its driving corridor 138 in amanner consistent with the teleoperations signals, for example, as shownin FIG. 7B. Upon expansion of the driving corridor 138, the vehicle 102may generate a plurality of revised trajectories based at least in parton the altered boundaries 140 of the driving corridor 138. The vehicle102 may calculate a revised confidence level for each of the revisedtrajectories, and may select the revised trajectory having the highestconfidence level from among the plurality of revised trajectories. Inthe example shown, the alert bar 536 indicates that the revisedconfidence level of the vehicle 102 is 95% by displaying “SystemConfidence 95%.” Based at least in part on the selected revisedtrajectory, the vehicle 102 may determine a revised drive line 146 foruse in maneuvering past the construction zone 702 including the staticobjects 704 and the dynamic object 706. Thereafter, the vehiclecontroller 228 may be configured to operate the vehicle 102 according tothe revised drive line 146, for example, as shown in FIG. 7B, andmaneuver past the construction zone 702.

In some examples, a teleoperator 150 may be able to use the live videofeed shown in the video view zone 528 to provide guidance to the vehicle102 in a zone such as the construction zone 702. For example, in aconstruction zone one or more construction workers or traffic police maybe present to direct traffic through or around the construction zone. Insome such examples, the vehicle 102 may send communication signals tothe teleoperations system 148 including live video signals, which may beviewed by the teleoperator 150 via the video view zone 528. Although inthe example shown, the video view zone 528 is confined to the upperright-hand corner of the example UIs 700A and 700B, in some examples,the teleoperator 150 may be able to reconfigured to UIs 700A and 700B sothat the video view zone 528 occupies a larger portion of the display.By viewing the live video, the teleoperator may be able to see theperson directing traffic and send teleoperations signals to the vehicle102 providing guidance to adhere to the direction of the persondirecting traffic, such that the vehicle controller 228 can maneuver thevehicle 102 through or around the construction zone.

FIG. 8A is an example UI 800A showing a fourth event scenario in whichthe vehicle 102 has encountered an example static object 802 in the road106. In the example shown, the static object 802 is trash (e.g., a ballof paper or other sheet material). As the vehicle 102 approaches thestatic object 802, the sensors 204 associated with the vehicle 102 maydetect the static object 802, and the vehicle 102 may determine thelocation of the static object 802 and classify it. As the vehicle 102approaches the static object 802, the vehicle 102 may calculate aconfidence level, which, due to the static object 802 in the road 106,may drop below a threshold minimum confidence level. As a result, thevehicle 102 may slow its speed or stop, and use its network interface234 to send communication signals to the teleoperations system 148providing sensor data and a request for guidance from the teleoperationssystem 148. The request may be inferred and/or determined by theteleoperations system 148, for example, as noted above. Theteleoperations system 148 may receive the communication signals, thesituation may be evaluated by a teleoperator 150 via the teleoperationsinterface 154, and the teleoperator 150 may send teleoperations signalsto the vehicle 102 via the teleoperations transmitter 306 providingguidance, for example, as described herein.

The example UI 800B shown in FIG. 8B shows example teleoperator 150interaction with the vehicle 102 to address the static object 802 shownin FIGS. 8A and 8B. The example interaction bar 534 indicates that theteleoperator 150 is engaged with the vehicle 102, depicted by the shaded“Engaged” icon. However, the teleoperator 150 may be able to determinefrom the communication signals received from the vehicle 102 that thestatic object 802 is trash that does not need to be avoided. As such,the teleoperator 150 may send teleoperations signals to the vehicle 102providing guidance proposing to ignore the static object 802. Based onthis proposal, the vehicle 102 may maintain its original drive line 146,or may adjust it only slightly without altering the boundaries 140 ofthe driving corridor 138, and continue to maneuver via the vehiclecontroller 228 according to the original drive line 146.

FIGS. 9A-9C show example UIs 900A-900C showing a fifth event scenario inwhich the vehicle 102 has encountered a roadside repair that includesboth an example static object 902 and an example dynamic object 904 atleast partially the lane 110A in the road 106. In the example shown, thestatic object 902 is a car, and the dynamic object 904 is a personconducting a repair, in this example, changing a tire. As the vehicle102 approaches the static and dynamic objects 902 and 904, the sensors204 associated with the vehicle 102 may detect one or both of them, andthe vehicle 102 may determine the locations of the static and dynamicobjects 902 and 904, classify them, and/or predict a possible trajectoryof the dynamic object 904. As the vehicle 102 approaches, it maycalculate a confidence level associated with a probability that thevehicle 102 will be able to successfully maneuver past the static anddynamic objects 902 and 904. As the confidence level drops below athreshold minimum confidence level, the vehicle 102 may slow its speedor stop, and use its network interface 234 to send communication signalsto the teleoperations system 148 providing sensor data and a request forguidance from the teleoperations system 148. As noted above, the requestmay be inferred and/or determined by the teleoperations system 148. Theteleoperations system 148 may receive the communication signals, thesituation may be evaluated by a teleoperator 150 via the teleoperationsinterface 154, and the teleoperator 150 may send teleoperations signalsto the vehicle 102 via the teleoperations transmitter 306 providingguidance.

In the example shown in FIG. 9A, the alert bar 536 of the UI 900Aindicates for the teleoperator 150 that the vehicle 102 has arrived atat least one static object and at least one dynamic object in the road106, which obstruct the drive line 146 of the vehicle 102, and that theconfidence levels associated with each of those identifications is 100%.In the example shown, one or more arrows 906 may provide an indicationof a predicted trajectory of the dynamic object 904 based on, forexample, its type and/or classification, and/or its prior and/or currenttrajectory. In this example, the dynamic object 904 is a person kneelingnext to the tire of the car, and thus, it is unlikely the person willmove farther into the road 106. In the example UI 900A shown in FIG. 9A,the active view zone 522 depicts the scenario as a schematic perspectiveanimation, and the video view zone 528 provides a live video of thestatic and dynamic objects 902 and 904. In some situations, it may beeasier for the teleoperator 150 to more quickly and/or more accuratelydetermine the nature of the event being encountered by the vehicle 102by viewing the video view zone 528. This may result in the teleoperator150 being able to provide the vehicle 102 with guidance more quicklyand/or more accurately.

FIG. 9B shows an example UI 900B providing an example of theteleoperator 150 providing guidance to the vehicle 102. The exampleinteraction bar 534 indicates that the teleoperator 150 is engaged withthe vehicle 102, depicted by the shaded “Engaged” icon, and that theteleoperator 150 is identifying an area for the vehicle 102 to avoid,depicted by the “Area-Avoid” icon being shaded. As shown, theteleoperator 150 has used the teleoperations input device to erect avirtual wall 908 around the static and dynamic objects 902 and 904 tonotify the vehicle 102 to avoid entering the area bounded by the virtualwall 908.

FIG. 9C an example UI 900C providing a revised driving corridor 138based on the area bounded by the virtual wall 908 shown in FIG. 9B. Inthe example shown, the teleoperator 150 has expanded the boundaries 140of the driving corridor 138 to the second lane 110, across the lanedividing line 114. In some examples, the teleoperator 150 may accomplishthis using the teleoperations input device 158, for example, as notedabove. Based on the teleoperator's inputs, the teleoperations system 148may transmit teleoperations signals to the vehicle 102 via theteleoperations transmitter 306. In the example shown, the vehicle 102may expand the boundaries 140 of its driving corridor 138 in a mannerconsistent with the teleoperations signals, for example, as shown inFIG. 9C. Upon expansion of the driving corridor 138, the vehicle 102 maygenerate a plurality of revised trajectories based at least in part onthe altered boundaries 140 of the driving corridor 138. The vehicle 102may calculate a confidence level for each of the revised trajectories,and may select a revised trajectory having the highest confidence levelfrom among the plurality of revised trajectories. Based at least in parton the selected revised trajectory, the vehicle 102 may determine arevised drive line 146 for use in maneuvering past the static anddynamic objects 902 and 904. Thereafter, the vehicle controller 228 maybe configured to operate the vehicle 102 according to the revised driveline 146, for example, as shown in FIG. 9C and maneuver past the event.

FIG. 10 is a schematic overhead view of an example road network 1000including three example vehicles 102A-102C en route between respectivefirst geographic locations 120A-120C and respective destinations124A-124C at second geographic areas. For example, a first vehicle 102Ais shown traveling along a planned path 1002A between a first geographiclocation 120A and a destination 124A. Second and third respectivevehicles 102B and 102C each travel along respective planned paths 1002Band 1002C between respective first geographic locations 120B and 120Cand respective destinations 124B and 124C. In the example shown, thefirst planned path 1002A passes an accident zone 1004, which may createa slow traffic condition and unique driving circumstances, such as laneclosures and merging traffic. The second planned path 1002B passesthrough a school zone 1006, which may present unique drivingcircumstances, such as a school zone speed limit, crosswalks, andsomeone directing traffic. The third planned path 1002C passes through aconstruction zone 1008, which may also present unique drivingconditions, such as lane changes and someone directing traffic.

The example first vehicle 102A may normally operate according to a firstoperating mode associated with first operating parameters via thevehicle controller 228 along the road network 1000 according to thefirst path 1002A. In some examples, the teleoperations system 148 mayreceive, via the teleoperations receiver 304, via at least one ofanother entity (e.g., a navigation- or traffic information-relatedentity) or a vehicle 102 other than the first vehicle 102A,communication signals indicating occurrence of an event associated witha second geographic area located along the first path 1002A. In thisexample, the event is the example accident zone 1004. Similarly, theteleoperations system 148 may receive, via the teleoperations receiver304, via at least one of another entity or a vehicle 102 other than thesecond and third vehicles 102B or 102C, communication signals indicatingoccurrence of respective events associated with respective secondgeographic areas located along the second and third paths 1002B and1002C. In the example shown, the respective events are the school andconstruction zones 1006 and 1008. A teleoperator 150 in communicationwith the teleoperations receiver 304 may evaluate data associated withthe communication signals and classify the geographic areas associatedwith the accident zone 1004, the school zone 1006, and/or theconstruction zone 1008 as corresponding to respective zones in which thevehicle controllers 228 of the respective first, second, and thirdvehicles 102A, 102B, and 102C operate the respective vehicles accordingto respective second operating modes associated with second operatingparameters, wherein at least one of the second operating parametersdiffers from a corresponding first operating parameter. For example, thesecond operating parameters for operating the first vehicle 102A whileit is in the vicinity of the accident zone 1004 may include operating ata slower speed, sending signals to the teleoperations system 148, sothat the teleoperations system 148 may provide guidance to assist thefirst vehicle 102A as it passes through the accident zone 1004, forexample, so that the first vehicle 102A can change lanes and/or complywith instructions given by someone directing traffic in the vicinity ofthe accident zone 1004. The second operating parameters for operatingthe second vehicle 102B while it is in the vicinity of the school zone1006 may include operating at a slower speed, sending signals to theteleoperations system 148, so that the teleoperations system 148 mayprovide guidance to assist the second vehicle 102B as it passes throughthe school zone 1006, for example, so that the second vehicle 102B canstop for crosswalks when people are present at the crosswalks and/orcomply with instructions given by someone directing traffic in theschool zone 1006. Similarly, the second operating parameters foroperating the third vehicle 102C while it is in the vicinity of theconstruction zone 1008 may include operating at a slower speed andsending signals to the teleoperations system 148, so that theteleoperations system 148 may provide guidance to assist the thirdvehicle 102C as it passes through the construction zone 1008, forexample, so that the third vehicle 102C can change lanes and/or complywith instructions given by someone directing traffic in the vicinity ofthe construction zone 1008. In some examples, the teleoperations system148 may send teleoperations signals via the teleoperations transmitter306, to the vehicles 102A-102C to provide guidance to the respectivevehicle controllers 228 of the vehicles 102A-102C to switch from thefirst operating mode to the second operating mode while operating in therespective second geographic areas. In some examples, the secondoperating parameters may include one or more of altered performanceparameters (e.g., speed, acceleration, braking rates, and steering inputrates), altered vehicle operation policies (e.g., safety-relatedguidelines for controlling the vehicle), altered vehicle operation laws,or vehicle operation regulations.

Although the example events described with respect to FIG. 10 includeaccident, school, and construction zones, other geographiclocation-related zones are contemplated. For example, other events maybe associated with flood zones, parade zones, special event zones,and/or zones associated with slow traffic, such as areas where vehiclesare being driven into bright sunlight or areas where weather conditionssuch as rain or snow are affecting traffic rates.

As mentioned previously herein, road network data may include datarelated to a global or local map of an area associated with operation ofthe vehicle 102. In some examples, the local and/or global map may beconfigured to be updated by another party using any information relatingto any occurrence of an event associated with a geographic area locatedalong a path on which the vehicle 102 travels. For example, a policedepartment may provide information to set policies/regulations foroperating in an area, construction workers may provide the scope of aproject for incorporation into the local and/or global map, etc.

In some examples, the teleoperations system 148 may be configured tosend teleoperations signals providing guidance to all or a subset of thevehicles 102 of the fleet 302. For example, the teleoperations system148 may be configured to send teleoperations signals to at least asubset of the vehicles 102 providing guidance to switch operation from afirst operating mode including first operating parameters to a secondoperating mode including second operating parameters, at least one ofwhich is different than a corresponding first operating parameter. Forexample, the second operating parameters may include one or more ofsecond performance parameters, second vehicle operation policies, secondvehicle operation laws, and second vehicle operation regulations. Insome examples, guidance may include switching from the first operatingmode to the second operating mode for a predetermined period of time andthereafter returning to the first operating mode. In some examples, thesecond operating parameters may include one or more of reducing energyexpenditure of the vehicles 102, setting a maximum operating speed,preventing the vehicles 102 from operating bidirectionally, changing athreshold confidence level required for autonomous operation, changingthe threshold confidence level required for autonomous operation in adefined geographic area, altering at least one of an objectclassification model or an object prediction model used by the vehicles,and relaxing vehicle operation policies associated with complying withtraffic laws and regulations.

In some examples, the teleoperations signals to each vehicle 102 of thesubset may provide guidance to the respective vehicle controllers 228 toavoid a geographic area based at least in part on a presence of an eventassociated with the road network 104, for example, as previouslydescribed. For example, the geographic area may correspond to one ormore of a construction zone, a school zone, a flood zone, an accidentzone, a parade zone, a special event zone, and a zone associated with aslow traffic condition, and the teleoperations signals may includeguidance for operating each of the vehicles 102 of the subset accordingto the second operating mode that corresponds to at least one of thezones.

The subset of vehicles 102 may include one or more of vehicles 102carrying at least one occupant, vehicles having no occupants, vehiclesincluding at least one battery having a charge below a threshold levelof charge, and vehicles configured to determine a status of conditionsassociated with the road network 104. For example, if a vehicle 102 iscarrying at least one occupant, the vehicle 102 may operate in a modethat favors a short and/or comfortable ride over a path having a shorterdistance, which might be preferred for a vehicle having at least onebattery below a threshold level of charge. A vehicle 102 being used todetermine the status of conditions associated with the road network maybe operated according to a mode that favors traveling at a certain speedover selected roads 106 of the road network 104 in order to determinethe status of the selected roads.

FIGS. 11-16 are flow diagrams of example processes illustrated as acollection of blocks in a logical flow graph, which represent a sequenceof operations that can be implemented in hardware, software, or acombination thereof. In the context of software, the blocks representcomputer-executable instructions stored on one or more computer-readablestorage media that, when executed by one or more processors, perform therecited operations. Generally, computer-executable instructions includeroutines, programs, objects, components, data structures, and the likethat perform particular functions or implement particular abstract datatypes. The order in which the operations are described is not intendedto be construed as a limitation, and any number of the described blockscan be combined in any order and/or in parallel to implement theprocesses.

FIG. 11 is a flow diagram of an example process 1100 for operating adriverless vehicle including a vehicle controller. At 1102, the exampleprocess 1100 may include receiving road network data from a road networkdata store. The road network data may be based at least in part on alocation of the driverless vehicle. In some examples, this may includeglobal and/or local map data that may be stored and/or updated by thedriverless vehicle and/or by the teleoperations system.

At 1104, the example process 1100 may include receiving, at thedriverless vehicle, sensor signals including sensor data from one ormore sensors associated with the driverless vehicle. In some examples,the sensor data may be related to operation of the driverless vehicle.For example, the sensor data may include sensor signals associated withthe environment through which the driverless vehicle is traveling.

At 1106, the example process 1100 may include determining, at thedriverless vehicle, a driving corridor within which the driverlessvehicle travels according to a trajectory. For example, the drivingcorridor may include virtual boundaries and/or may be based at least inpart on one or more of the sensor data or the road network data.

The example process 1100, at 1108, may further include causing thedriverless vehicle to traverse a road network autonomously according toa path from a first geographic location to a second geographic locationdifferent than the first geographic location. For example, thedriverless vehicle may travel without the assistance of a driver betweena starting point, along the path, to the destination. In some examples,following 1108, the process may include returning to 1102 and repeating1102-1108, for example, until an event associated with the path has beendetermined, as outlined below.

At 1110, the example process 1100 may include determining that an eventassociated with the path has occurred. As explained herein, the eventmay be any condition along the path that may cause the confidence levelof the driverless vehicle to drop below a threshold confidence level.

At 1112, the example process 1100 may include sending communicationsignals from the driverless vehicle to a teleoperations system. Forexample, the communication signals may include a request for guidancefrom the teleoperations system and at least one of the sensor data orthe road network data.

At 1114, the example process 1100 may further include receiving, at thedriverless vehicle, teleoperations signals from the teleoperationssystem. For example, the teleoperations signals may include guidance toalter the virtual boundaries of the driving corridor, such that thevehicle controller determines a revised trajectory. For example, thevehicle controller may generate a plurality of revised trajectories, forexample, concurrently or substantially simultaneously (within technicalcapabilities), based at least in part on the altered virtual boundariesof the driving corridor. Each of the revised trajectories may beassociated with a confidence level, and the example process 1100 mayfurther include selecting a revised trajectory having the highestconfidence level from among the plurality of revised trajectories, andthe driverless vehicle may be operated according to the selected revisedtrajectory.

FIG. 12 is a flow diagram of an example process 1200 for operating adriverless vehicle, with the driverless vehicle including a vehiclecontroller and autonomously operating according to a first operatingmode associated with first operating parameters via the vehiclecontroller along a road network according to a path from a firstgeographic location to a destination separated from the first geographiclocation. At 1202, the example process 1200 may include receiving, via ateleoperations receiver located remotely from the driverless vehicle,via at least one of another entity or the driverless vehicle,communication signals indicating occurrence of an event associated witha second geographic area located along the path.

At 1204, the example process 1200 may include reviewing, by ateleoperator in communication with the teleoperations receiver, sensordata associated with sensor signals received from one more sensorsassociated with the driverless vehicle. For example, the sensor data maybe related to operation of the driverless vehicle.

The example process 1200, at 1206, may include classifying, via the atleast one of the other entity or the teleoperator, the second geographicarea as corresponding to a zone in which the vehicle controller operatesthe driverless vehicle according to a second operating mode associatedwith second operating parameters. In some examples, one or more of thesecond operating parameters may differ from a corresponding firstoperating parameter, for example, as discussed herein.

At 1208, the example process 1200 may also include sendingteleoperations signals, via a teleoperations transmitter, to thedriverless vehicle. The teleoperations signals may provide guidance tothe vehicle controller to switch from the first operating mode to thesecond operating mode while operating in the second geographic area. Thesecond operating parameters may include at least one of secondperformance parameters, second vehicle operation policies, secondvehicle operation laws, or second vehicle operation regulations. Thesecond geographic area may correspond to one or more of a constructionzone, a school zone, a flood zone, an accident zone, a parade zone, aspecial event zone, or a zone associated with a slow traffic condition.

FIG. 13 is a flow diagram of an example process 1300 for alteringoperation of at least a subset of a plurality of driverless vehicles ofa fleet of driverless vehicles. Each driverless vehicle of the subsetmay include a vehicle controller and may autonomously operate accordingto a first operating mode associated with first operating parameters viathe vehicle controller along a road network according to a respectivepath from a respective first geographic location to a respectivedestination separated from the first geographic location. At 1302, theexample process 1300 may include operating a plurality of driverlessvehicles of a fleet of driverless vehicles according to the firstoperating mode. The example process 1300, at 1304, may include sendingteleoperations signals, via a teleoperations transmitter incommunication with a teleoperator and located remotely from thedriverless vehicles, to each driverless vehicle of the subset, theteleoperations signals providing guidance to the respective vehiclecontrollers to switch from the first operating mode to a secondoperating mode. In some examples, the second operating mode may beassociated with second operating parameters including one or more ofsecond performance parameters, second vehicle operation policies, secondvehicle operation laws, and second vehicle operation regulations, and atleast one of the second operating parameters may differ from acorresponding first operating parameter. At 1306, the example process1300 may include operating the subset of driverless vehicles of thefleet of driverless vehicles according to the second operating mode.

FIG. 14 is a flow diagram for an example process 1400 for operating adriverless vehicle, with the driverless vehicle including a vehiclecontroller and autonomously operating via the vehicle controller along aroad network according to a path from a first geographic location to adestination separated from the first geographic location. At 1402, theexample process 1400 may include receiving, via a teleoperationsreceiver located remotely from the driverless vehicle, communicationsignals from the driverless vehicle. The communication signals mayinclude sensor data from one or more sensors associated with operationof the driverless vehicle. The communication signals may also includedata indicating occurrence of an event associated with the path, whereinthe data indicating occurrence of the event includes data indicating aconfidence level associated with the path is less than a thresholdconfidence level.

At 1404, the example process 1400 may also include reviewing, via ateleoperator in communication with the teleoperations receiver, thesensor data and the data indicating occurrence of the event. At 1406,the example process 1400 may include determining, via the teleoperator,a level of guidance to provide the driverless vehicle based at least inpart on one or more of the sensor data and the data indicatingoccurrence of the event. In some examples, the level of guidance mayrelate to whether the teleoperator provides instructions to thedriverless vehicle, collaborates with the driverless vehicle, forexample, trading information and/or proposed actions, or confirmsinformation received from the driverless vehicle and/or actions proposedby the driverless vehicle.

At 1408, the example process 1400 may further include sendingteleoperations signals, via a teleoperations transmitter, to thedriverless vehicle. The teleoperations signals may include guidance tooperate the driverless vehicle according to the determined level ofguidance, so that the vehicle controller maneuvers the driverlessvehicle to at least one of avoid the event, travel around the event, orpass through the event. At 1410, the example process 1400 may alsoinclude controlling the maneuvering of the driverless vehicle via thevehicle controller to avoid, travel around, and/or pass through theevent.

In some examples, the event may include an object impeding completion ofa portion of the path, and sending the teleoperations signals mayinclude sending teleoperations signals including a proposed trajectoryfor use by the vehicle controller to avoid the object to permit thevehicle controller to maneuver the driverless vehicle past the object.In some examples, the teleoperations signals may include guidance in theform of teleoperations signals that provide a proposed reclassificationof the object and/or that confirm a classification of the object. Suchreclassification may result in a different predicted trajectory of theobject, which may increase the confidence level to a level at which thedriverless vehicle can continue along its original drive line and/ortrajectory. In some examples, the communication signals from thedriverless vehicle may include data related to classification of theobject, and the teleoperations signals may propose ignoring the objectto permit the vehicle controller to maneuver the driverless vehicle pastthe object. For example, if the object is trash or a small stick thatwould not pose a safety or operational problem, the driverless vehiclemay simply not attempt to avoid the object. In some examples, thecommunication signals from the driverless vehicle may include a proposedtrajectory for passing the object and completing the portion of thepath. In such examples, the teleoperations signals may confirm ordecline the proposed trajectory, and thus, the teleoperations system maymerely authorize the trajectory proposed by the driverless vehicle. Insome examples, when the teleoperations signals decline the proposedtrajectory, the teleoperations signals may also provide an alternativeproposed trajectory for the vehicle controller to maneuver thedriverless vehicle past the object.

In some examples, the communication signals from the driverless vehiclemay include a proposal to sound an audible warning, a proposal toactivate lights to provide a visual warning, and/or a proposal to movethe driverless vehicle slowly forward. In such examples, theteleoperations signals may accept or decline at least one of theproposals. For example, the object may be an unattended animal withinthe driving corridor, which will cause the driverless vehicle slow orstop. The driverless vehicle may use an audible warning and/or activatelights to encourage the animal to exit the road, and/or the vehicle mayinch forward slowly to encourage the animal to exit the road.

In some examples, the event may include an operational rule preventingcompletion of a portion of the path, and the teleoperations signals maypropose a modification to the operational rule to permit the vehiclecontroller to maneuver the driverless vehicle past the event. Forexample, the road may include a shoulder and use of the shoulder wouldenable the driverless vehicle to travel past the object. However,operational rules of the driverless vehicle may prevent the driverlessfrom using the shoulder. In some such situations, the teleoperationssignals may propose using the shoulder.

In some examples, the event may be the driverless vehicle lackinginformation sufficient to complete a portion of the path, and theteleoperations signals may provide information sufficient to permit thevehicle controller to maneuver the driverless vehicle past the event.For example, the driverless vehicle may not be able to identify and/orclassify an object, and teleoperations signals may identify or classifythe object so that the driverless vehicle is able to take appropriateaction. Alternatively, the teleoperations signals may provide a proposedtrajectory for the vehicle controller to maneuver the driverless vehiclepast the event.

In some examples, the communication signals from the driverless vehiclemay include a predicted trajectory of an object into a path of thedriverless vehicle. In some such circumstances, the teleoperationssignals may alter the predicted trajectory, so that the vehiclecontroller maneuvers the driverless vehicle past the object.

FIG. 15 is a flow diagram of an example process 1500 for operating adriverless vehicle. The driverless vehicle may include a vehiclecontroller and may autonomously operate via the vehicle controller alonga road network according to a path from a first geographic location to adestination separated from the first geographic location. At 1502, theexample process 1500 may include receiving, via a teleoperationsreceiver located remotely from the driverless vehicle, firstcommunication signals from the driverless vehicle. The firstcommunication signals may include first sensor data related to operationof the driverless vehicle from one or more sensors associated with thedriverless vehicle. The first communication signals may also includedata indicating occurrence of a first event associated with the path.The first event may include first characteristics including at least onecharacteristic not previously encountered by the driverless vehicle orat least one characteristic previously encountered by the driverlessvehicle fewer than a threshold number of occurrences. The firstcommunication signals may also include a request for guidance to passthe event and continue along the path.

At 1504, the example process 1500 may also include reviewing, via ateleoperator in communication with the teleoperations receiver, dataassociated with the first communication signals received from thedriverless vehicle. For example, the teleoperator may use ateleoperations interface to facilitate this review.

At 1506, the example process 1500 may also include determining, via theteleoperator, a first level of guidance for providing the driverlessvehicle based at least in part on the data associated with the firstcommunication signals. For example, the first level of guidance mayrelate to whether the teleoperator provides instructions to thedriverless vehicle, collaborates with the driverless vehicle, forexample, trading information and/or proposed actions, or confirmsinformation received from the driverless vehicle and/or actions proposedby the driverless vehicle.

At 1508, the example process 1500 may include sending firstteleoperations signals, via a teleoperations transmitter, to thedriverless vehicle. The first teleoperations signals may include thefirst level of guidance, so that the vehicle controller maneuvers thedriverless vehicle to pass the first event and continue along the pathaccording to the first level of guidance.

At 1510, the example process 1500 may further include receiving, via theteleoperations receiver, second communication signals from thedriverless vehicle. The second communication signals may include secondsensor data related to operation of the driverless vehicle from one ormore sensors associated with the driverless vehicle. The secondcommunication signals may also include data indicating occurrence of asecond event associated with the path. The second event may includesecond characteristics, wherein the second characteristics include atleast one second characteristic in common with at least one of the firstcharacteristics. The second communication signals may also include arequest for at least one of information related to the second event or aproposed action for passing the second event and continuing along thepath.

At 1512, the example process 1500 may also include reviewing, via ateleoperator in communication with the teleoperations receiver dataassociated with the second communication signals received from thedriverless vehicle. At 1514, the example process 1500 may also includedetermining, via the teleoperator, a second level of guidance forproviding the driverless vehicle based at least in part on the dataassociated with the second communication signals.

At 1516, the example process 1500 may also include sending secondteleoperations signals, via the teleoperations transmitter, to thedriverless vehicle. In some examples, the second teleoperations signalsmay include the second level of guidance, and the second level ofguidance may include at least one of the information related to thesecond event or the proposed second action, so that the vehiclecontroller maneuvers the driverless vehicle to pass the second event andcontinue along the path based at least in part on at least one of theinformation related to the second event or the proposed second action.

In some examples of the process 1500, the process may further includereceiving, in a machine learning engine including an event responsemodel, data associated with the first communication signals and/or thedata associated with the second communication signals. In some suchexamples, the event response model may be updated based at least in parton the data associated with the first communication signals, the dataassociated with the second communication signals, the first level ofguidance, and/or the second level of guidance. In this example manner,the process 1500 may improve over time with the updates. For example,the process 1500 may also include transforming the first level ofguidance into the second level of guidance based at least in part on theupdated event response model.

FIG. 16 is a flow diagram of an example process 1600 for operating aplurality of driverless vehicles. The driverless vehicles may eachinclude a vehicle controller and may autonomously operate via thevehicle controller along a road network according to a path from a firstgeographic location to a destination separated from the first geographiclocation.

At 1602, the example process 1600 may include receiving, via ateleoperations receiver located remotely from a first one of theplurality of driverless vehicles, first communication signals from thefirst driverless vehicle indicating occurrence of a first eventassociated with the road network along a path associated with the firstdriverless vehicle. The first event may include first characteristics,and the first communication signals may include a request for guidanceto pass the event and continue along the path.

At 1604, the example process 1600 may include reviewing, via a firstteleoperator in communication with the teleoperations receiver, dataassociated with the first communication signals received from the firstdriverless vehicle. At 1606, the example process 1600 may also includedetermining, via the first teleoperator, a first level of guidance forproviding the first driverless vehicle based at least in part on atleast one of the data associated with the first communication signals,and at 1608, sending first teleoperations signals, via a teleoperationstransmitter, to the first driverless vehicle, wherein the firstteleoperations signals include the first level of guidance.

At 1610, the example process 1600 may also include receiving, via ateleoperations receiver, second communication signals from a seconddriverless vehicle of the plurality of driverless vehicles indicatingoccurrence of a second event associated with the road network along apath associated with the second driverless vehicle. The second event mayinclude second characteristics, wherein the second characteristicsinclude at least one second characteristic in common with at least oneof the first characteristics, and the second communication signals mayinclude a request for information related to the second event and/or aproposed action for passing the second event and continuing along thepath.

At 1612, the example process 1600 may further include reviewing, via ateleoperator in communication with a teleoperations receiver, dataassociated with the second communication signals received from thesecond driverless vehicle, and at 1614, determining, via theteleoperator, a second level of guidance for providing the seconddriverless vehicle based at least in part on data associated with thefirst event and the data associated with the second communicationsignals.

At 1616, the example process 1600 may also include sending secondteleoperations signals, via the teleoperations transmitter, to thesecond driverless vehicle. In some examples, the second teleoperationssignals may include the second level of guidance, wherein the secondlevel of guidance includes the information related to the second eventand/or the proposed second action, so that the vehicle controllermaneuvers the second driverless vehicle to pass the second event andcontinue along the path based at least in part on at least one of theinformation related to the second event or the proposed second action.

It should be appreciated that the subject matter presented herein may beimplemented as a computer process, a computer-controlled apparatus, acomputing system, or an article of manufacture, such as acomputer-readable storage medium. While the subject matter describedherein is presented in the general context of modules that may includehardware and/or software layers that execute on one or more computingdevices, those skilled in the art will recognize that otherimplementations may be performed in combination with other types ofprogram modules. Generally, software modules include routines, programs,components, data structures, and other types of structures that performparticular tasks or implement particular abstract data types. In someexamples, any number of modules (hardware and/or software) may beemployed, and techniques described herein as employed by one or moremodules may be employed by a greater or lesser number of modules.

Those skilled in the art will also appreciate that aspects of thesubject matter described herein may be practiced on or in conjunctionwith other computer system configurations beyond those described herein,including multiprocessor systems, microprocessor-based or programmableconsumer electronics, minicomputers, mainframe computers, handheldcomputers, mobile telephone devices, tablet computing devices,special-purposed hardware devices, network appliances, and the like.

Although the subject matter presented herein has been described inlanguage specific to computer structural features, methodological acts,and computer readable media, it is to be understood that the inventiondefined in the appended claims is not necessarily limited to thespecific features, acts, or media described herein. Rather, the specificfeatures, acts, and media are disclosed as example forms of implementingthe subject matter recited in the claims.

The subject matter described above is provided by way of illustrationonly and should not be construed as limiting. Furthermore, the claimedsubject matter is not limited to implementations that solve any or alldisadvantages noted in any part of this disclosure. Variousmodifications and changes may be made to the subject matter describedherein without following the examples and applications illustrated anddescribed, and without departing from the spirit and scope of thepresent invention, which is set forth in the following claims.

What is claimed is:
 1. A method comprising: receiving sensor data fromone or more sensors associated with a vehicle; determining, based atleast in part on a location of the vehicle, environment data associatedwith an environment of the vehicle; determining, based at least in parton the sensor data or the environment data, a driving corridorcomprising a virtual boundary within which the vehicle travels accordingto a trajectory; determining that an event occurs in the environment ofthe vehicle; sending a request for guidance to a teleoperation system;receiving, from the teleoperation system, a teleoperation signal, theteleoperation signal including one or more commands associated with thevirtual boundary; and controlling the vehicle based at least in part onthe teleoperation signal.
 2. The method of claim 1, wherein theteleoperation signal comprises a first command to: alter, as an alteredvirtual boundary, the virtual boundary of the driving corridor; andoperate the vehicle based at least in part on the altered virtualboundary.
 3. The method of claim 2, wherein the teleoperation signalcomprises a second command to: generate a plurality of proposedtrajectories based at least in part on the altered virtual boundary,each proposed trajectory of the plurality of proposed trajectories beingassociated with a confidence level; determine, as a revised trajectory,an individual proposed trajectory having a highest confidence level fromamong the plurality of proposed trajectories; and operate the vehicle totravel according to the revised trajectory.
 4. The method of claim 3,wherein the teleoperation signal comprises a request to confirm ordecline the revised trajectory.
 5. The method of claim 4, furthercomprising: receiving an input to decline the revised trajectory;determining an alternative revised trajectory from the plurality ofproposed trajectories; and operating the vehicle to travel according tothe alternative revised trajectory.
 6. The method of claim 3, whereinthe teleoperation signal comprises a command to: operate the vehicle totravel along the revised trajectory to overcome the event and to returnto the trajectory after overcoming the event.
 7. The method of claim 1,wherein determining that the event occurs in the environment of thevehicle comprises: determining first classification data associated withan object impeding the trajectory, wherein the teleoperation signalcomprises second classification data associated with the object that isdifferent from the first classification data and a command to ignore theobject, based at least in part on the second classification data, topermit the vehicle to travel past the object.
 8. A system comprising:one or more processors; and one or more non-transitory computer readablemedia storing computer executable instructions that, when executed,cause the one or more processors to perform operations comprising:receiving sensor data from one or more sensors associated with avehicle; determining, based at least in part on a location of thevehicle, environment data associated with an environment of the vehicle;determining, based on at least one of the sensor data or the environmentdata, a driving corridor comprising a virtual boundary within which thevehicle travels according to a trajectory; determining that an eventoccurs in the environment of the vehicle; sending a request for guidanceto a teleoperation system; receiving, from the teleoperation system, ateleoperation signal, the teleoperation signal including one or morecommands associated with the virtual boundary; and controlling thevehicle based at least in part on the teleoperation signal.
 9. Thesystem of claim 8, wherein the teleoperation signal comprises a commandto: operate the vehicle to travel along a revised trajectory to overcomethe event and to continue along the trajectory after overcoming theevent.
 10. The system of claim 8, wherein determining that the eventoccurs in the environment of the vehicle comprises: determining firstclassification data associated with an object impeding the trajectory,wherein the teleoperation signal comprises second classification dataassociated with the object that is different from the firstclassification data and a command to ignore the object, based at leastin part on the second classification data, to permit the vehicle totravel past the object.
 11. The system of claim 8, wherein theteleoperation signal comprises a first command to: alter, as an alteredvirtual boundary, the virtual boundary of the driving corridor; andoperate the vehicle based at least in part on the altered virtualboundary.
 12. The system of claim 11, wherein the teleoperation signalcomprises a second command to: generate a plurality of proposedtrajectories based at least in part on the altered virtual boundary,each proposed trajectory of the plurality of proposed trajectories beingassociated with a confidence level; determine, as a revised trajectory,an individual proposed trajectory having a highest confidence level fromamong the plurality of proposed trajectories; and operate the vehicle totravel according to the revised trajectory.
 13. The system of claim 12,wherein the teleoperation signal comprises a request to confirm ordecline the revised trajectory.
 14. The system of claim 13, theoperations further comprising: receiving an input to decline the revisedtrajectory; determining an alternative revised trajectory from theplurality of proposed trajectories; and operating the vehicle to travelaccording to the alternative revised trajectory.
 15. One or morenon-transitory computer-readable media storing instructions that, whenexecuted, cause one or more processors to perform operations comprising:receiving sensor data from one or more sensors associated with avehicle; determining, based at least in part on a location of thevehicle, environment data associated with an environment of the vehicle;determining, based on at least one of the sensor data or the environmentdata, a driving corridor comprising a virtual boundary within which thevehicle travels according to a trajectory; determining that an eventoccurs in the environment of the vehicle; sending a request for guidanceto a teleoperation system; receiving, from the teleoperation system, ateleoperation signal, the teleoperation signal including one or morecommands associated with the virtual boundary; and controlling thevehicle based at least in part on the teleoperation signal.
 16. The oneor more non-transitory computer-readable media of claim 15, wherein theteleoperation signal comprises a first command to: alter, as an alteredvirtual boundary, the virtual boundary of the driving corridor; andoperate the vehicle based at least in part on the altered virtualboundary.
 17. The one or more non-transitory computer-readable media ofclaim 16, wherein the teleoperation signal comprises a second commandto: generate a plurality of proposed trajectories based at least in parton the altered virtual boundary, each proposed trajectory of theplurality of proposed trajectories being associated with a confidencelevel; determine, as a revised trajectory, an individual proposedtrajectory having a highest confidence level from among the plurality ofproposed trajectories; and operate the vehicle to travel according tothe revised trajectory.
 18. The one or more non-transitorycomputer-readable media of claim 17, wherein the teleoperation signalcomprises a request to confirm or decline the revised trajectory. 19.The one or more non-transitory computer-readable media of claim 18,wherein the operations further comprise: receiving an input to declinethe revised trajectory; determining an alternative revised trajectoryfrom the plurality of proposed trajectories; and operating the vehicleto travel according to the alternative revised trajectory.
 20. The oneor more non-transitory computer-readable media of claim 15, whereindetermining that the event occurs in the environment of the vehiclecomprises: determining first classification data associated with anobject impeding the trajectory, wherein the teleoperation signalcomprises second classification data associated with the object that isdifferent from the first classification data and a command to ignore theobject, based at least in part on the second classification data, topermit the vehicle to travel past the object.